Jul 28 17:10:42.938007 ip-10-0-139-66 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:10:43.294045 ip-10-0-139-66 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:43.294045 ip-10-0-139-66 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:10:43.294045 ip-10-0-139-66 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:43.294045 ip-10-0-139-66 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:10:43.294656 ip-10-0-139-66 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:43.295773 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.295661    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:10:43.302657 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302632    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:43.302657 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302651    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:43.302657 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302655    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:43.302657 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302659    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:43.302657 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302662    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:43.302657 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302667    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302670    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302673    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302676    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302679    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302682    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302685    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302688    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302690    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302693    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302696    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302699    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302702    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302704    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302707    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302710    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302728    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302731    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302734    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302737    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:43.302922 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302740    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302742    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302746    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302749    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302751    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302754    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302762    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302765    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302768    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302770    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302773    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302775    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302780    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302785    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302789    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302793    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302796    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302799    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302802    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:43.303401 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302805    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302808    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302810    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302813    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302816    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302820    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302823    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302825    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302828    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302831    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302833    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302836    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302839    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302842    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302846    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302849    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302852    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302854    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302857    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:43.303921 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302860    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302863    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302865    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302868    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302870    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302873    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302875    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302878    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302882    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302885    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302888    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302892    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302897    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302900    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302903    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302906    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302909    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302912    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302916    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:43.304381 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302918    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302928    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302931    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302934    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302937    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302940    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302942    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302945    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302948    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302950    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302953    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302955    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302958    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302961    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302963    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302966    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.302968    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303747    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303754    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303757    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:43.304869 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303760    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303764    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303767    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303770    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303772    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303775    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303778    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303780    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303783    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303785    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303788    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303791    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303793    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303796    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303806    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303809    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303811    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303814    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303816    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303819    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:43.305354 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303823    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303827    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303831    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303834    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303837    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303840    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303843    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303845    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303848    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303850    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303853    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303855    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303858    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303862    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303866    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303868    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303871    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303874    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303876    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303879    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:43.305861 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303881    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303884    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303886    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303889    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303891    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303894    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303896    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303906    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303909    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303912    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303915    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303918    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303920    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303923    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303926    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303928    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303931    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303934    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303936    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303939    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:43.306349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303941    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303944    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303948    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303951    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303954    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303957    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303960    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303962    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303965    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303967    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303970    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303973    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303975    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303978    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303981    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303983    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303986    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303988    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303992    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:43.306844 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.303995    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304004    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304007    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304009    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304012    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304015    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304018    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304020    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304022    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304025    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304027    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304030    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304032    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304035    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304037    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304040    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304043    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304441    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304447    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304450    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:43.307300 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304452    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304455    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304458    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304461    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304464    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304467    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304469    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304472    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304474    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304477    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304480    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304482    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304485    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304493    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304496    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304498    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304501    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304503    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304506    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304508    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:43.307827 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304511    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304515    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304518    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304521    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304524    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304526    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304528    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304531    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304534    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304536    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304539    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304542    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304545    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304547    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304550    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304552    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304555    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304557    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304560    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:43.308307 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304563    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304565    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304568    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304570    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304573    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304575    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304578    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304586    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304588    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304591    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304593    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304596    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304598    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304601    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304603    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304606    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304609    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304612    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304615    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:43.308889 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304617    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304619    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304622    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304625    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304628    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304630    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304633    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304636    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304638    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304640    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304643    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304645    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304648    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304650    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304652    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304656    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304659    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304664    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304667    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304670    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:43.309373 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304673    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304681    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304683    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304686    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304688    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304691    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304693    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304696    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304698    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304701    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304703    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304705    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304708    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304710    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304727    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304732    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304735    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.304738    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304803    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304810    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304824    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:10:43.309879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304831    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304837    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304842    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304847    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304852    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304855    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304858    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304861    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304865    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304868    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304876    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304879    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304882    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304892    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304895    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304903    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304906    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304910    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304913    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304916    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304920    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304923    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304925    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:10:43.310437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304928    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304931    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304934    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304937    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304941    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304944    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304948    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304951    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304953    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304956    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304959    2578 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304962    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304969    2578 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304972    2578 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304975    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304978    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304981    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304985    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304988    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304992    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304995    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.304998    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305001    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305009    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305012    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:10:43.311087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305015    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305018    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305021    2578 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305028    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305031    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305034    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305037    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305040    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305043    2578 flags.go:64] FLAG: --help="false"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305046    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305049    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305052    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305055    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305058    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305062    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305064    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305067    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305084    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305088    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305091    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305094    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305097    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305100    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305102    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:10:43.311741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305106    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305108    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305113    2578 flags.go:64] FLAG: --lock-file=""
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305116    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305119    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305122    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305127    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305136    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305139    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305142    2578 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305145    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305148    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305151    2578 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305153    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305158    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305161    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305165    2578 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305167    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305174    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305177    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305180    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305183    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305186    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305189    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305196    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:10:43.312326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305199    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305202    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305205    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305208    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305212    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305215    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305218    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305221    2578 flags.go:64] FLAG: --port="10250"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305224    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305229    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0460f6ac0146e5843"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305232    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305235    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305238    2578 flags.go:64] FLAG: --register-node="true"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305241    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305244    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305253    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305256    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305259    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305263    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305266    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305269    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305272    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305274    2578 flags.go:64] FLAG: --runonce="false"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305277    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305280    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:10:43.312965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305283    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305287    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305290    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305293    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305296    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305299    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305302    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305305    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305308    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305310    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305314    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305316    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305319    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305324    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305327    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305330    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305338    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305341    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305344    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305347    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305350    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305353    2578 flags.go:64] FLAG: --v="2"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305357    2578 flags.go:64] FLAG: --version="false"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305367    2578 flags.go:64] FLAG: --vmodule=""
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305374    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:10:43.313568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305378    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305529    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305533    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305536    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305539    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305542    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305544    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305547    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305552    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305554    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305557    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305561    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305563    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305566    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305569    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305573    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305576    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305579    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305582    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305585    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:43.314228 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305587    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305590    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305592    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305597    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305599    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305602    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305605    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305607    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305609    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305612    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305615    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305623    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305625    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305628    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305631    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305633    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305636    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305639    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305641    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305644    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:43.314737 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305647    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305650    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305652    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305655    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305657    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305660    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305662    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305665    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305668    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305670    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305672    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305675    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305678    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305680    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305683    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305686    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305689    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305691    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305694    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305696    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:43.315227 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305699    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305701    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305704    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305706    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305729    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305733    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305736    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305738    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305741    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305745    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305749    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305751    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305756    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305761    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305765    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305768    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305771    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305774    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305776    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:43.315792 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305779    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305781    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305784    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305786    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305789    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305792    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305794    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305797    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305801    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305803    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305806    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305808    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305811    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305813    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305816    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305819    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305821    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305824    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305834    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305837    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:43.316286 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.305839    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.305844    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.312146    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.312161    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.312611    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.312619    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312682    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312687    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312690    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312693    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312696    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312699    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312702    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312704    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312707    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:43.316911 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312709    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312726    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312731    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312735    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312737    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312740    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312743    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312747    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312751    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312753    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312757    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312759    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312762    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312764    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312767    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312770    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312772    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312775    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312777    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:43.317279 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312780    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312783    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312786    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312789    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312791    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312794    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312796    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312799    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312801    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312805    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312809    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312812    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312815    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312818    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312821    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312823    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312826    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312828    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312831    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312834    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:43.317838 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312836    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312838    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312841    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312844    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312846    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312849    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312851    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312854    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312856    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312858    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312861    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312864    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312866    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312868    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312872    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312875    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312877    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312880    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312883    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:43.318612 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312885    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312888    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312891    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312893    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312896    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312898    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312901    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312903    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312906    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312908    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312911    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312914    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312916    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312919    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312922    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312924    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312927    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312929    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312932    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:43.319339 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312934    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312937    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312939    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312943    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312946    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312949    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312952    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312955    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312957    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312961    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312964    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312967    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.312970    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.312975    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313099    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:43.319950 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313103    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313107    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313111    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313114    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313117    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313120    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313122    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313125    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313128    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313131    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313134    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313137    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313140    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313143    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313145    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313148    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313150    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313152    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313155    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:43.320553 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313157    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313160    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313162    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313165    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313167    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313169    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313172    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313175    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313179    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313182    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313184    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313187    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313190    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313194    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313197    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313200    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313203    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313205    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313208    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:43.321027 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313212    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313215    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313217    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313220    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313223    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313225    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313228    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313230    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313233    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313235    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313238    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313240    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313243    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313245    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313248    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313250    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313253    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313255    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313258    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313260    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:43.321484 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313263    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313266    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313269    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313272    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313274    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313277    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313279    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313282    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313284    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313287    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313289    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313292    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313294    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313296    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313299    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313301    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313304    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313307    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313309    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313312    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:43.322006 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313315    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313317    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313322    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313324    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313327    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313329    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313332    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313334    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313337    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313340    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313344    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313347    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313350    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313353    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313356    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313359    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313361    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313364    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:43.322541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:43.313367    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:43.323007 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.313372    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:43.323007 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.313495    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:10:43.323007 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.315240    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:10:43.323007 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.316239    2578 server.go:1019] "Starting client certificate rotation"
Jul 28 17:10:43.323007 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.316354    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:43.323007 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.316396    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:43.323007 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.318368    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:43.325568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.325547    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:43.334322 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.334305    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:10:43.336762 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.336749    2578 log.go:25] "Validated CRI v1 image API"
Jul 28 17:10:43.338602 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.338582    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:10:43.341436 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.341418    2578 fs.go:135] Filesystem UUIDs: map[0c6e5c4f-7cf4-4e6f-9d1a-65620266520c:/dev/nvme0n1p4 58149f7a-446a-4caf-b2ef-345927a42689:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 17:10:43.341499 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.341436    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:10:43.346434 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.346321    2578 manager.go:216] Machine: {Timestamp:2026-07-28 17:10:43.345104994 +0000 UTC m=+0.298324115 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3149162 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d9b5cf66d5d3fe17c9a39eb27f21 SystemUUID:ec27d9b5-cf66-d5d3-fe17-c9a39eb27f21 BootID:6fa4dab0-6bdb-4dbe-bf61-50e72d852d41 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:1d:ff:45:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:1d:ff:45:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:f4:15:22:94:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:10:43.346434 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.346428    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:10:43.346571 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.346500    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:10:43.349482 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.349452    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:10:43.349614 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.349483    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:10:43.350097 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.350086    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:10:43.350140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.350099    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:10:43.350140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.350107    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:43.350707 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.350696    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:43.351642 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.351630    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:10:43.354512 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.354499    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:43.354632 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.354622    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:10:43.355206 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.355193    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:10:43.355247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.355219    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:10:43.355247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.355234    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:10:43.355247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.355243    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:10:43.355360 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.355252    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:10:43.355615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.355603    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:43.355655 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.355624    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:43.357757 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.357741    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:10:43.359233 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.359212    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:10:43.359297 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.359249    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:10:43.361488 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361383    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:10:43.361607 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361590    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:10:43.361658 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361613    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:10:43.361658 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361623    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:10:43.361658 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361636    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:10:43.361658 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361647    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:10:43.361869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361661    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:10:43.361869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361671    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:10:43.361869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361684    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:10:43.361869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361693    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:10:43.361869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361711    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:10:43.361869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.361739    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:10:43.362582 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.362560    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:43.362904 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.362889    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:10:43.363008 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.362999    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:10:43.364275 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.364248    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:10:43.364340 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.364250    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:10:43.364645 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.364630    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:10:43.366420 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.366401    2578 server.go:1268] "Started kubelet"
Jul 28 17:10:43.366803 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.366504    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:10:43.366803 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.366604    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:10:43.366803 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.366603    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:10:43.367196 ip-10-0-139-66 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:10:43.367959 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.367944    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:10:43.368731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.368705    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:10:43.371361 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.371344    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:10:43.371840 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.371034    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c68347060c48e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.366365409 +0000 UTC m=+0.319584529,LastTimestamp:2026-07-28 17:10:43.366365409 +0000 UTC m=+0.319584529,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.373168 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.373082    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:10:43.373168 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.373085    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:43.373405 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.373216    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:43.373668 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.373527    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:10:43.373668 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.373540    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:10:43.373789 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.373687    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:10:43.373842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.373794    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:10:43.373842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.373803    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:10:43.374460 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.374441    2578 factory.go:55] Registering systemd factory
Jul 28 17:10:43.374460 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.374463    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:10:43.376517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.375634    2578 factory.go:153] Registering CRI-O factory
Jul 28 17:10:43.376517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.375650    2578 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:10:43.376517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.375707    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:10:43.376517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.375743    2578 factory.go:103] Registering Raw factory
Jul 28 17:10:43.376517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.375756    2578 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:10:43.376517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.376141    2578 manager.go:349] Starting recovery of all containers
Jul 28 17:10:43.384647 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.384620    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:10:43.384739 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.384657    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:10:43.389422 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.389396    2578 manager.go:354] Recovery completed
Jul 28 17:10:43.394589 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.394574    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:43.396410 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.396386    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:43.396485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.396419    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:43.396485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.396434    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:43.396992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.396978    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:10:43.396992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.396983    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:43.396992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.396990    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:10:43.397154 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.397005    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:43.397694 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.397676    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:43.397770 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.397728    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:43.397770 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.397743    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:43.399070 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.399000    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c6834707d69d1b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.396402459 +0000 UTC m=+0.349621586,LastTimestamp:2026-07-28 17:10:43.396402459 +0000 UTC m=+0.349621586,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.399602 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.399590    2578 policy_none.go:49] "None policy: Start"
Jul 28 17:10:43.399642 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.399606    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:10:43.399642 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.399616    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:10:43.401626 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.401615    2578 policy_none.go:47] "Start"
Jul 28 17:10:43.407642 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.407553    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c6834707d6f895  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.396425877 +0000 UTC m=+0.349644998,LastTimestamp:2026-07-28 17:10:43.396425877 +0000 UTC m=+0.349644998,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.416699 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.416547    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c6834707d72d23  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.396439331 +0000 UTC m=+0.349658455,LastTimestamp:2026-07-28 17:10:43.396439331 +0000 UTC m=+0.349658455,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.425976 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.425919    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-66.ec2.internal.18c6834707d69d1b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c6834707d69d1b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.396402459 +0000 UTC m=+0.349621586,LastTimestamp:2026-07-28 17:10:43.397695697 +0000 UTC m=+0.350914828,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.433932 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.433875    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-66.ec2.internal.18c6834707d6f895\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c6834707d6f895  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.396425877 +0000 UTC m=+0.349644998,LastTimestamp:2026-07-28 17:10:43.397735854 +0000 UTC m=+0.350954974,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.446338 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.446325    2578 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:10:43.446419 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.446354    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:10:43.446419 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.446364    2578 server.go:87] "Starting device plugin registration server"
Jul 28 17:10:43.446577 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.446565    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:10:43.446655 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.446584    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:10:43.446753 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.446734    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:10:43.446818 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.446805    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:10:43.446818 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.446815    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:10:43.447282 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.447265    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:10:43.447371 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.447316    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:43.447371 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.447236    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-66.ec2.internal.18c6834707d72d23\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c6834707d72d23  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-66.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.396439331 +0000 UTC m=+0.349658455,LastTimestamp:2026-07-28 17:10:43.397748481 +0000 UTC m=+0.350967602,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.458907 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.458846    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.ec2.internal.18c683470af07d6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.ec2.internal,UID:ip-10-0-139-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.ec2.internal,},FirstTimestamp:2026-07-28 17:10:43.448429932 +0000 UTC m=+0.401649042,LastTimestamp:2026-07-28 17:10:43.448429932 +0000 UTC m=+0.401649042,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.ec2.internal,}"
Jul 28 17:10:43.514359 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.514335    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6mm56"
Jul 28 17:10:43.522983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.522967    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6mm56"
Jul 28 17:10:43.547573 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.547517    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:43.548403 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.548389    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:43.548460 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.548415    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:43.548460 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.548439    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:43.548528 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.548473    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.564479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.564460    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.564546 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.564482    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-66.ec2.internal\": node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:43.568666 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.568646    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:10:43.569837 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.569818    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:10:43.569926 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.569853    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:10:43.569926 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.569874    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:10:43.569926 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.569909    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:10:43.571655 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.571634    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:43.580810 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.580793    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:43.670423 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.670390    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"]
Jul 28 17:10:43.670522 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.670485    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:43.672229 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.672215    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:43.672297 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.672240    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:43.672297 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.672249    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:43.674467 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.674454    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:43.674538 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.674466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e68a05610ec5fd1eced0e51b5c0733c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"e68a05610ec5fd1eced0e51b5c0733c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.674538 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.674489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e68a05610ec5fd1eced0e51b5c0733c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"e68a05610ec5fd1eced0e51b5c0733c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.674653 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.674628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.674698 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.674684    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:43.675207 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.675191    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:43.675276 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.675218    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:43.675276 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.675229    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:43.675619 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.675596    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:43.675711 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.675627    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:43.675711 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.675642    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:43.677806 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.677791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.677879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.677816    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:43.678419 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.678402    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:43.678502 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.678440    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:43.678502 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:43.678456    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:43.680901 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.680883    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:43.698214 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.698195    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-66.ec2.internal\" not found" node="ip-10-0-139-66.ec2.internal"
Jul 28 17:10:43.702438 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:43.702423    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-66.ec2.internal\" not found" node="ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.562772 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.562755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e68a05610ec5fd1eced0e51b5c0733c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"e68a05610ec5fd1eced0e51b5c0733c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.562842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.562788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e68a05610ec5fd1eced0e51b5c0733c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"e68a05610ec5fd1eced0e51b5c0733c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.562842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.562811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfcf7e829014a4c1c014b92346f6e0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"cfcf7e829014a4c1c014b92346f6e0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.562923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.562845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e68a05610ec5fd1eced0e51b5c0733c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"e68a05610ec5fd1eced0e51b5c0733c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.562923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.562886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e68a05610ec5fd1eced0e51b5c0733c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal\" (UID: \"e68a05610ec5fd1eced0e51b5c0733c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.568864 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:45.568849    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:45.663469 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.663408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfcf7e829014a4c1c014b92346f6e0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"cfcf7e829014a4c1c014b92346f6e0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.663469 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.663447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfcf7e829014a4c1c014b92346f6e0a6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.ec2.internal\" (UID: \"cfcf7e829014a4c1c014b92346f6e0a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.669538 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:45.669517    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:45.770054 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:45.770018    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:45.789187 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.789167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.792424 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:45.792411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:45.870992 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:45.870953    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:45.971519 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:45.971459    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:46.071990 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.071947    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-66.ec2.internal\" not found"
Jul 28 17:10:46.092377 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.092354    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:46.104187 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.104168    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:10:46.104285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.104269    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:46.104350 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.104326    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:46.144567 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.144539    2578 apiserver.go:52] "Watching apiserver"
Jul 28 17:10:46.161120 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.161096    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:46.161581 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.161567    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:46.164583 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.164565    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:10:46.164903 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.164884    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-gf2g6","openshift-ovn-kubernetes/ovnkube-node-qdr2l","kube-system/global-pull-secret-syncer-9dlb9","openshift-image-registry/node-ca-2kxnh","openshift-multus/multus-additional-cni-plugins-hxxn4","openshift-multus/multus-g757g","openshift-multus/network-metrics-daemon-rr4wq","openshift-network-operator/iptables-alerter-fcc6d","kube-system/konnectivity-agent-2d7hz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd","openshift-cluster-node-tuning-operator/tuned-b78kd","openshift-dns/node-resolver-v7x9l"]
Jul 28 17:10:46.167436 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.167410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:46.167499 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.167481    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:10:46.169493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.169478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.170141 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.170126    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:46.170195 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.170179    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"
Jul 28 17:10:46.171571 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.171558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.171625 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.171608    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:10:46.172351 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.172336    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:10:46.172677 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.172655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.172921 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.172903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:10:46.173012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.172926    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-czz62\""
Jul 28 17:10:46.173804 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.173766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.174078 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.173966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:10:46.174078 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.173984    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.174078 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.174043    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:10:46.175997 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.175984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.177946 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.177930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.178142 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.178125    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:10:46.178218 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.178131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.178218 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.178197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vtdfb\""
Jul 28 17:10:46.178566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.178551    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.178877 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.178865    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:10:46.179032 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.179020    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.179556 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.179540    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tgdmh\""
Jul 28 17:10:46.179634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.179568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.179634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.179580    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:10:46.179634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.179580    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:10:46.180355 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.180341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:46.180433 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.180413    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:10:46.180959 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.180945    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-th2l2\""
Jul 28 17:10:46.181146 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.181130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:10:46.182617 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.182601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.185667 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.184984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.185667 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.185009    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:46.187175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.185930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:10:46.187175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.186117    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.187175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.186215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g2scz\""
Jul 28 17:10:46.187175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.186223    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.188249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.187735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.188249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.188107    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:10:46.189087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.189070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-889pl\""
Jul 28 17:10:46.189591 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.189575    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:10:46.190064 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.190052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.191001 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.190970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:10:46.191056 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.191006    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.191238 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.191226    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xmhq9\""
Jul 28 17:10:46.191542 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.191520    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.192156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.192138    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal"]
Jul 28 17:10:46.192246 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.192164    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal"]
Jul 28 17:10:46.192246 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.192234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.193082 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.193067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.193584 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.193569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hbj9q\""
Jul 28 17:10:46.193644 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.193607    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.195202 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.195188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.195262 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.195249    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tkgwg\""
Jul 28 17:10:46.195316 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.195261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.223129 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.223086    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:46.223231 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.223171    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcrb8"
Jul 28 17:10:46.228759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.228743    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcrb8"
Jul 28 17:10:46.262290 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.262270    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:10:46.265941 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.265922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-kubernetes\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266033 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.265945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-cni-bin\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266033 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.265961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.266033 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.265980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-hostroot\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.266033 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-conf-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.266258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.266258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-os-release\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.266258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-systemd\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-var-lib-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-ovn\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysctl-d\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-systemd\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-lib-modules\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vszrl\" (UniqueName: \"kubernetes.io/projected/112143bf-d024-4e4a-837a-c6c4b6421010-kube-api-access-vszrl\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-kubelet\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-run-netns\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/112143bf-d024-4e4a-837a-c6c4b6421010-etc-tuned\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-cni-netd\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-cni-multus\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnl4l\" (UniqueName: \"kubernetes.io/projected/4fc84898-1712-4173-8e8a-3eb68293e69d-kube-api-access-mnl4l\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:46.266535 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysconfig\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-var-lib-kubelet\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwr5x\" (UniqueName: \"kubernetes.io/projected/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-kube-api-access-wwr5x\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-ovnkube-script-lib\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-system-cni-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-cnibin\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-log-socket\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysctl-conf\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-systemd-units\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-slash\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-os-release\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-run\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-host\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-serviceca\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-system-cni-dir\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.266884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4da92ddf-8155-41a3-bce0-e723bc0787c0-agent-certs\") pod \"konnectivity-agent-2d7hz\" (UID: \"4da92ddf-8155-41a3-bce0-e723bc0787c0\") " pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f339ccd3-739e-4820-bbd6-1909f519163c-tmp-dir\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-k8s-cni-cncf-io\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-daemon-config\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d45de308-c906-41dd-91e0-b5d65ad0434e-host-slash\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112143bf-d024-4e4a-837a-c6c4b6421010-tmp\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-cnibin\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.266982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkm9\" (UniqueName: \"kubernetes.io/projected/af516b76-8f7c-494c-9c49-d0b92829113d-kube-api-access-gzkm9\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-kubelet\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-etc-kubernetes\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-sys-fs\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-node-log\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-env-overrides\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-cni-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-sys\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f339ccd3-739e-4820-bbd6-1909f519163c-hosts-file\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.267479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-etc-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-ovnkube-config\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30185e23-4967-457e-836b-9a335ab75449-kubelet-config\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-cni-binary-copy\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-device-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8h2\" (UniqueName: \"kubernetes.io/projected/f339ccd3-739e-4820-bbd6-1909f519163c-kube-api-access-dj8h2\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af516b76-8f7c-494c-9c49-d0b92829113d-ovn-node-metrics-cert\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30185e23-4967-457e-836b-9a335ab75449-dbus\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-cni-bin\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4da92ddf-8155-41a3-bce0-e723bc0787c0-konnectivity-ca\") pod \"konnectivity-agent-2d7hz\" (UID: \"4da92ddf-8155-41a3-bce0-e723bc0787c0\") " pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-registration-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-modprobe-d\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2985\" (UniqueName: \"kubernetes.io/projected/f1b81611-bb9f-4327-8b30-27fdf4699404-kube-api-access-g2985\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-socket-dir-parent\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.267992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-multus-certs\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79zj\" (UniqueName: \"kubernetes.io/projected/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-kube-api-access-k79zj\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhqr\" (UniqueName: \"kubernetes.io/projected/e994bd0c-54f8-4539-a96e-59083cc6d258-kube-api-access-hjhqr\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-host\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-socket-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-netns\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d45de308-c906-41dd-91e0-b5d65ad0434e-iptables-alerter-script\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.268496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.267718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmslv\" (UniqueName: \"kubernetes.io/projected/d45de308-c906-41dd-91e0-b5d65ad0434e-kube-api-access-qmslv\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.272171 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.272153    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:46.316604 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.314868    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:05:45 +0000 UTC" deadline="2028-04-22 19:23:26.43423583 +0000 UTC"
Jul 28 17:10:46.316604 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.314963    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15218h12m40.119276696s"
Jul 28 17:10:46.368070 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/112143bf-d024-4e4a-837a-c6c4b6421010-etc-tuned\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-cni-netd\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-cni-multus\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-cni-netd\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-cni-multus\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.368216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnl4l\" (UniqueName: \"kubernetes.io/projected/4fc84898-1712-4173-8e8a-3eb68293e69d-kube-api-access-mnl4l\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysconfig\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-var-lib-kubelet\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysconfig\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwr5x\" (UniqueName: \"kubernetes.io/projected/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-kube-api-access-wwr5x\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-ovnkube-script-lib\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-system-cni-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-cnibin\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368423    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-log-socket\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysctl-conf\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-system-cni-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-systemd-units\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-var-lib-kubelet\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-slash\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-cnibin\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.368601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-os-release\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-run\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-host\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-systemd-units\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-serviceca\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysctl-conf\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.368626    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-slash\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-os-release\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.368696    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs podName:4fc84898-1712-4173-8e8a-3eb68293e69d nodeName:}" failed. No retries permitted until 2026-07-28 17:10:46.86867477 +0000 UTC m=+2.034310607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs") pod "network-metrics-daemon-rr4wq" (UID: "4fc84898-1712-4173-8e8a-3eb68293e69d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-system-cni-dir\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-log-socket\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-host\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-run\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-system-cni-dir\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.369492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4da92ddf-8155-41a3-bce0-e723bc0787c0-agent-certs\") pod \"konnectivity-agent-2d7hz\" (UID: \"4da92ddf-8155-41a3-bce0-e723bc0787c0\") " pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f339ccd3-739e-4820-bbd6-1909f519163c-tmp-dir\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-k8s-cni-cncf-io\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-daemon-config\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d45de308-c906-41dd-91e0-b5d65ad0434e-host-slash\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112143bf-d024-4e4a-837a-c6c4b6421010-tmp\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-cnibin\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.368994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkm9\" (UniqueName: \"kubernetes.io/projected/af516b76-8f7c-494c-9c49-d0b92829113d-kube-api-access-gzkm9\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-kubelet\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-etc-kubernetes\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-serviceca\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-sys-fs\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-node-log\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-env-overrides\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-ovnkube-script-lib\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-cni-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-sys\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-k8s-cni-cncf-io\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.370354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f339ccd3-739e-4820-bbd6-1909f519163c-hosts-file\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-etc-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-sys-fs\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-ovnkube-config\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30185e23-4967-457e-836b-9a335ab75449-kubelet-config\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369273    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-cnibin\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-cni-binary-copy\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-device-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8h2\" (UniqueName: \"kubernetes.io/projected/f339ccd3-739e-4820-bbd6-1909f519163c-kube-api-access-dj8h2\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-kubelet\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af516b76-8f7c-494c-9c49-d0b92829113d-ovn-node-metrics-cert\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-etc-kubernetes\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30185e23-4967-457e-836b-9a335ab75449-dbus\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d45de308-c906-41dd-91e0-b5d65ad0434e-host-slash\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-cni-bin\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-etc-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.371223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4da92ddf-8155-41a3-bce0-e723bc0787c0-konnectivity-ca\") pod \"konnectivity-agent-2d7hz\" (UID: \"4da92ddf-8155-41a3-bce0-e723bc0787c0\") " pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f339ccd3-739e-4820-bbd6-1909f519163c-tmp-dir\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-registration-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-sys\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-modprobe-d\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-cni-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2985\" (UniqueName: \"kubernetes.io/projected/f1b81611-bb9f-4327-8b30-27fdf4699404-kube-api-access-g2985\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-node-log\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-modprobe-d\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-socket-dir-parent\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-multus-certs\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79zj\" (UniqueName: \"kubernetes.io/projected/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-kube-api-access-k79zj\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhqr\" (UniqueName: \"kubernetes.io/projected/e994bd0c-54f8-4539-a96e-59083cc6d258-kube-api-access-hjhqr\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-host\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.372453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-socket-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-netns\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-daemon-config\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d45de308-c906-41dd-91e0-b5d65ad0434e-iptables-alerter-script\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmslv\" (UniqueName: \"kubernetes.io/projected/d45de308-c906-41dd-91e0-b5d65ad0434e-kube-api-access-qmslv\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-kubernetes\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-cni-bin\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-var-lib-cni-bin\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-hostroot\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30185e23-4967-457e-836b-9a335ab75449-dbus\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-conf-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-host\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.369543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f339ccd3-739e-4820-bbd6-1909f519163c-hosts-file\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-os-release\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-systemd\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.372923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-var-lib-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-env-overrides\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-ovn\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-multus-certs\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.370320    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.370384    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret podName:30185e23-4967-457e-836b-9a335ab75449 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:46.870368016 +0000 UTC m=+2.036003838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret") pod "global-pull-secret-syncer-9dlb9" (UID: "30185e23-4967-457e-836b-9a335ab75449") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-socket-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af516b76-8f7c-494c-9c49-d0b92829113d-ovnkube-config\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-host-run-netns\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30185e23-4967-457e-836b-9a335ab75449-kubelet-config\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-registration-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-device-dir\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-hostroot\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-kubernetes\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-conf-dir\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.373457 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e994bd0c-54f8-4539-a96e-59083cc6d258-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-systemd\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1b81611-bb9f-4327-8b30-27fdf4699404-os-release\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-var-lib-openvswitch\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-cni-bin\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-cni-binary-copy\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.370979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d45de308-c906-41dd-91e0-b5d65ad0434e-iptables-alerter-script\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-run-ovn\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-multus-socket-dir-parent\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysctl-d\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-systemd\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-lib-modules\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszrl\" (UniqueName: \"kubernetes.io/projected/112143bf-d024-4e4a-837a-c6c4b6421010-kube-api-access-vszrl\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-systemd\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.373975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-kubelet\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-run-netns\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-lib-modules\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/112143bf-d024-4e4a-837a-c6c4b6421010-etc-sysctl-d\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4da92ddf-8155-41a3-bce0-e723bc0787c0-konnectivity-ca\") pod \"konnectivity-agent-2d7hz\" (UID: \"4da92ddf-8155-41a3-bce0-e723bc0787c0\") " pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-run-netns\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af516b76-8f7c-494c-9c49-d0b92829113d-host-kubelet\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.371849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1b81611-bb9f-4327-8b30-27fdf4699404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.372144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/112143bf-d024-4e4a-837a-c6c4b6421010-etc-tuned\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.372194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112143bf-d024-4e4a-837a-c6c4b6421010-tmp\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.372231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af516b76-8f7c-494c-9c49-d0b92829113d-ovn-node-metrics-cert\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.374486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.373861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4da92ddf-8155-41a3-bce0-e723bc0787c0-agent-certs\") pod \"konnectivity-agent-2d7hz\" (UID: \"4da92ddf-8155-41a3-bce0-e723bc0787c0\") " pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.382852 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.382828    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:46.382960 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.382855    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:46.382960 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.382869    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9gj4f for pod openshift-network-diagnostics/network-check-target-gf2g6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:46.382960 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.382925    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f podName:9ec6705e-d32d-48c8-851f-0c353cf1463e nodeName:}" failed. No retries permitted until 2026-07-28 17:10:46.882908902 +0000 UTC m=+2.048544740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9gj4f" (UniqueName: "kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f") pod "network-check-target-gf2g6" (UID: "9ec6705e-d32d-48c8-851f-0c353cf1463e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:46.384367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.383891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwr5x\" (UniqueName: \"kubernetes.io/projected/1a2cb418-91da-4de5-a28e-ed1cf6ffea5f-kube-api-access-wwr5x\") pod \"node-ca-2kxnh\" (UID: \"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f\") " pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.384367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.384118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnl4l\" (UniqueName: \"kubernetes.io/projected/4fc84898-1712-4173-8e8a-3eb68293e69d-kube-api-access-mnl4l\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:46.384367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.384329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmslv\" (UniqueName: \"kubernetes.io/projected/d45de308-c906-41dd-91e0-b5d65ad0434e-kube-api-access-qmslv\") pod \"iptables-alerter-fcc6d\" (UID: \"d45de308-c906-41dd-91e0-b5d65ad0434e\") " pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.384854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.384834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79zj\" (UniqueName: \"kubernetes.io/projected/c91bf743-8bcf-47c4-b0b5-90c088cbbd26-kube-api-access-k79zj\") pod \"multus-g757g\" (UID: \"c91bf743-8bcf-47c4-b0b5-90c088cbbd26\") " pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.385285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.385243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhqr\" (UniqueName: \"kubernetes.io/projected/e994bd0c-54f8-4539-a96e-59083cc6d258-kube-api-access-hjhqr\") pod \"aws-ebs-csi-driver-node-9f4cd\" (UID: \"e994bd0c-54f8-4539-a96e-59083cc6d258\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.385658 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.385614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszrl\" (UniqueName: \"kubernetes.io/projected/112143bf-d024-4e4a-837a-c6c4b6421010-kube-api-access-vszrl\") pod \"tuned-b78kd\" (UID: \"112143bf-d024-4e4a-837a-c6c4b6421010\") " pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.386004 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.385981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2985\" (UniqueName: \"kubernetes.io/projected/f1b81611-bb9f-4327-8b30-27fdf4699404-kube-api-access-g2985\") pod \"multus-additional-cni-plugins-hxxn4\" (UID: \"f1b81611-bb9f-4327-8b30-27fdf4699404\") " pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.386201 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.386172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkm9\" (UniqueName: \"kubernetes.io/projected/af516b76-8f7c-494c-9c49-d0b92829113d-kube-api-access-gzkm9\") pod \"ovnkube-node-qdr2l\" (UID: \"af516b76-8f7c-494c-9c49-d0b92829113d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.386273 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.386253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8h2\" (UniqueName: \"kubernetes.io/projected/f339ccd3-739e-4820-bbd6-1909f519163c-kube-api-access-dj8h2\") pod \"node-resolver-v7x9l\" (UID: \"f339ccd3-739e-4820-bbd6-1909f519163c\") " pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.405811 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.405789    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcf7e829014a4c1c014b92346f6e0a6.slice/crio-dffdcc56bdf8e20d43ce43059cd382bb22b19d0f6cb5cf0717e2430afa8b0ad1 WatchSource:0}: Error finding container dffdcc56bdf8e20d43ce43059cd382bb22b19d0f6cb5cf0717e2430afa8b0ad1: Status 404 returned error can't find the container with id dffdcc56bdf8e20d43ce43059cd382bb22b19d0f6cb5cf0717e2430afa8b0ad1
Jul 28 17:10:46.406189 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.406171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68a05610ec5fd1eced0e51b5c0733c5.slice/crio-875253d172f67085bd4686ac880b6e16a722588094c205802b67f9e097829050 WatchSource:0}: Error finding container 875253d172f67085bd4686ac880b6e16a722588094c205802b67f9e097829050: Status 404 returned error can't find the container with id 875253d172f67085bd4686ac880b6e16a722588094c205802b67f9e097829050
Jul 28 17:10:46.411128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.411111    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:10:46.424674 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.424656    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:46.504176 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.504123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:10:46.512625 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.512606    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf516b76_8f7c_494c_9c49_d0b92829113d.slice/crio-d95427bb0125304fbef8e97f288e08ed868da1c8cf6ed1f9e6fff684f7c5ee0a WatchSource:0}: Error finding container d95427bb0125304fbef8e97f288e08ed868da1c8cf6ed1f9e6fff684f7c5ee0a: Status 404 returned error can't find the container with id d95427bb0125304fbef8e97f288e08ed868da1c8cf6ed1f9e6fff684f7c5ee0a
Jul 28 17:10:46.524499 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.524483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kxnh"
Jul 28 17:10:46.532183 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.532165    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2cb418_91da_4de5_a28e_ed1cf6ffea5f.slice/crio-d3d380024811ab105b5fe87493ca17e838c0b427ed76650fe7a4dea4df02a2d1 WatchSource:0}: Error finding container d3d380024811ab105b5fe87493ca17e838c0b427ed76650fe7a4dea4df02a2d1: Status 404 returned error can't find the container with id d3d380024811ab105b5fe87493ca17e838c0b427ed76650fe7a4dea4df02a2d1
Jul 28 17:10:46.544374 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.544356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hxxn4"
Jul 28 17:10:46.552134 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.552112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b81611_bb9f_4327_8b30_27fdf4699404.slice/crio-4b9dee0f0de79dc1915ebe393df24b9344e72643eaa7e88315d01574464e37b1 WatchSource:0}: Error finding container 4b9dee0f0de79dc1915ebe393df24b9344e72643eaa7e88315d01574464e37b1: Status 404 returned error can't find the container with id 4b9dee0f0de79dc1915ebe393df24b9344e72643eaa7e88315d01574464e37b1
Jul 28 17:10:46.557121 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.557105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g757g"
Jul 28 17:10:46.562820 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.562804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcc6d"
Jul 28 17:10:46.564427 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.564407    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91bf743_8bcf_47c4_b0b5_90c088cbbd26.slice/crio-339296d1de0fa36b72faf8e704531a17313ac52bca08dd5e2b74310c7d69e535 WatchSource:0}: Error finding container 339296d1de0fa36b72faf8e704531a17313ac52bca08dd5e2b74310c7d69e535: Status 404 returned error can't find the container with id 339296d1de0fa36b72faf8e704531a17313ac52bca08dd5e2b74310c7d69e535
Jul 28 17:10:46.570975 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.570955    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45de308_c906_41dd_91e0_b5d65ad0434e.slice/crio-de228afa0875a76cee9e6d5e400718ec1f15221e98e4e5eca5d5b7c82298a8f1 WatchSource:0}: Error finding container de228afa0875a76cee9e6d5e400718ec1f15221e98e4e5eca5d5b7c82298a8f1: Status 404 returned error can't find the container with id de228afa0875a76cee9e6d5e400718ec1f15221e98e4e5eca5d5b7c82298a8f1
Jul 28 17:10:46.577458 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.577442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:10:46.584634 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.584606    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da92ddf_8155_41a3_bce0_e723bc0787c0.slice/crio-3e9790de93bb9003b729be658536d0468b24d4809472cde164f39d486ea4b522 WatchSource:0}: Error finding container 3e9790de93bb9003b729be658536d0468b24d4809472cde164f39d486ea4b522: Status 404 returned error can't find the container with id 3e9790de93bb9003b729be658536d0468b24d4809472cde164f39d486ea4b522
Jul 28 17:10:46.593848 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.593830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd"
Jul 28 17:10:46.601910 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.601890    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode994bd0c_54f8_4539_a96e_59083cc6d258.slice/crio-54667e565837d6d4530f57a9f56432cc85b39953a1c368a3d94b02be6c199a54 WatchSource:0}: Error finding container 54667e565837d6d4530f57a9f56432cc85b39953a1c368a3d94b02be6c199a54: Status 404 returned error can't find the container with id 54667e565837d6d4530f57a9f56432cc85b39953a1c368a3d94b02be6c199a54
Jul 28 17:10:46.613039 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.613023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b78kd"
Jul 28 17:10:46.619595 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.619574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v7x9l"
Jul 28 17:10:46.620880 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.620861    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112143bf_d024_4e4a_837a_c6c4b6421010.slice/crio-0da53df4820bd24648798082925084443cbbce2114ff3344b1b402733160d5c2 WatchSource:0}: Error finding container 0da53df4820bd24648798082925084443cbbce2114ff3344b1b402733160d5c2: Status 404 returned error can't find the container with id 0da53df4820bd24648798082925084443cbbce2114ff3344b1b402733160d5c2
Jul 28 17:10:46.626965 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:10:46.626944    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf339ccd3_739e_4820_bbd6_1909f519163c.slice/crio-b275f8a1f0b9b8a1a3295466e7ab02225be1d118e044f9bc1177314ce8613814 WatchSource:0}: Error finding container b275f8a1f0b9b8a1a3295466e7ab02225be1d118e044f9bc1177314ce8613814: Status 404 returned error can't find the container with id b275f8a1f0b9b8a1a3295466e7ab02225be1d118e044f9bc1177314ce8613814
Jul 28 17:10:46.875748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.875652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:46.875748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.875733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:46.875941 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.875849    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:46.875941 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.875913    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs podName:4fc84898-1712-4173-8e8a-3eb68293e69d nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.87589551 +0000 UTC m=+3.041531334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs") pod "network-metrics-daemon-rr4wq" (UID: "4fc84898-1712-4173-8e8a-3eb68293e69d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:46.876203 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.875849    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:46.876366 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.876259    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret podName:30185e23-4967-457e-836b-9a335ab75449 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.876237167 +0000 UTC m=+3.041872999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret") pod "global-pull-secret-syncer-9dlb9" (UID: "30185e23-4967-457e-836b-9a335ab75449") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:46.976923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:46.976883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:46.977093 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.977080    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:46.977152 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.977102    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:46.977152 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.977115    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9gj4f for pod openshift-network-diagnostics/network-check-target-gf2g6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:46.977246 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:46.977174    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f podName:9ec6705e-d32d-48c8-851f-0c353cf1463e nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.977156082 +0000 UTC m=+3.142791912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gj4f" (UniqueName: "kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f") pod "network-check-target-gf2g6" (UID: "9ec6705e-d32d-48c8-851f-0c353cf1463e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:47.232828 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.232659    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:46 +0000 UTC" deadline="2028-03-26 18:33:13.32184722 +0000 UTC"
Jul 28 17:10:47.232828 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.232737    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14569h22m26.089126327s"
Jul 28 17:10:47.289383 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.289352    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:47.360006 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.359414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:47.360006 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.359534    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:10:47.374075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.374013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b78kd" event={"ID":"112143bf-d024-4e4a-837a-c6c4b6421010","Type":"ContainerStarted","Data":"0da53df4820bd24648798082925084443cbbce2114ff3344b1b402733160d5c2"}
Jul 28 17:10:47.386140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.386108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcc6d" event={"ID":"d45de308-c906-41dd-91e0-b5d65ad0434e","Type":"ContainerStarted","Data":"de228afa0875a76cee9e6d5e400718ec1f15221e98e4e5eca5d5b7c82298a8f1"}
Jul 28 17:10:47.393024 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.392994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g757g" event={"ID":"c91bf743-8bcf-47c4-b0b5-90c088cbbd26","Type":"ContainerStarted","Data":"339296d1de0fa36b72faf8e704531a17313ac52bca08dd5e2b74310c7d69e535"}
Jul 28 17:10:47.399228 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.399201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerStarted","Data":"4b9dee0f0de79dc1915ebe393df24b9344e72643eaa7e88315d01574464e37b1"}
Jul 28 17:10:47.405624 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.405585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"d95427bb0125304fbef8e97f288e08ed868da1c8cf6ed1f9e6fff684f7c5ee0a"}
Jul 28 17:10:47.412376 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.412338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" event={"ID":"cfcf7e829014a4c1c014b92346f6e0a6","Type":"ContainerStarted","Data":"dffdcc56bdf8e20d43ce43059cd382bb22b19d0f6cb5cf0717e2430afa8b0ad1"}
Jul 28 17:10:47.423158 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.423135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"e68a05610ec5fd1eced0e51b5c0733c5","Type":"ContainerStarted","Data":"875253d172f67085bd4686ac880b6e16a722588094c205802b67f9e097829050"}
Jul 28 17:10:47.430384 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.430356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v7x9l" event={"ID":"f339ccd3-739e-4820-bbd6-1909f519163c","Type":"ContainerStarted","Data":"b275f8a1f0b9b8a1a3295466e7ab02225be1d118e044f9bc1177314ce8613814"}
Jul 28 17:10:47.434029 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.434004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd" event={"ID":"e994bd0c-54f8-4539-a96e-59083cc6d258","Type":"ContainerStarted","Data":"54667e565837d6d4530f57a9f56432cc85b39953a1c368a3d94b02be6c199a54"}
Jul 28 17:10:47.439437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.439413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2d7hz" event={"ID":"4da92ddf-8155-41a3-bce0-e723bc0787c0","Type":"ContainerStarted","Data":"3e9790de93bb9003b729be658536d0468b24d4809472cde164f39d486ea4b522"}
Jul 28 17:10:47.453339 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.453311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kxnh" event={"ID":"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f","Type":"ContainerStarted","Data":"d3d380024811ab105b5fe87493ca17e838c0b427ed76650fe7a4dea4df02a2d1"}
Jul 28 17:10:47.884401 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.884359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:47.884600 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.884438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:47.884600 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.884584    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:47.884717 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.884646    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs podName:4fc84898-1712-4173-8e8a-3eb68293e69d nodeName:}" failed. No retries permitted until 2026-07-28 17:10:49.884627075 +0000 UTC m=+5.050262890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs") pod "network-metrics-daemon-rr4wq" (UID: "4fc84898-1712-4173-8e8a-3eb68293e69d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:47.885056 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.885035    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:47.885146 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.885100    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret podName:30185e23-4967-457e-836b-9a335ab75449 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:49.88508239 +0000 UTC m=+5.050718225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret") pod "global-pull-secret-syncer-9dlb9" (UID: "30185e23-4967-457e-836b-9a335ab75449") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:47.980162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.980129    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:47.986068 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:47.985435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:47.986068 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.985645    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:47.986068 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.985666    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:47.986068 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.985679    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9gj4f for pod openshift-network-diagnostics/network-check-target-gf2g6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:47.986068 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:47.985730    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f podName:9ec6705e-d32d-48c8-851f-0c353cf1463e nodeName:}" failed. No retries permitted until 2026-07-28 17:10:49.985713317 +0000 UTC m=+5.151349136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gj4f" (UniqueName: "kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f") pod "network-check-target-gf2g6" (UID: "9ec6705e-d32d-48c8-851f-0c353cf1463e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:48.233720 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:48.233631    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:46 +0000 UTC" deadline="2028-04-13 10:48:29.914660552 +0000 UTC"
Jul 28 17:10:48.233720 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:48.233671    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14993h37m41.68099456s"
Jul 28 17:10:48.358471 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:48.357715    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:48.358471 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:48.357734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:48.358471 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:48.357840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:10:48.358471 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:48.358257    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:10:49.360688 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:49.360649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:49.361118 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:49.360777    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:10:49.902365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:49.901551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:49.902365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:49.901631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:49.902365 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:49.901791    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:49.902365 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:49.901851    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs podName:4fc84898-1712-4173-8e8a-3eb68293e69d nodeName:}" failed. No retries permitted until 2026-07-28 17:10:53.901832623 +0000 UTC m=+9.067468435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs") pod "network-metrics-daemon-rr4wq" (UID: "4fc84898-1712-4173-8e8a-3eb68293e69d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:49.902365 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:49.902256    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:49.902365 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:49.902326    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret podName:30185e23-4967-457e-836b-9a335ab75449 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:53.902310648 +0000 UTC m=+9.067946474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret") pod "global-pull-secret-syncer-9dlb9" (UID: "30185e23-4967-457e-836b-9a335ab75449") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:50.002412 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:50.002374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:50.002619 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:50.002568    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:50.002619 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:50.002590    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:50.002619 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:50.002603    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9gj4f for pod openshift-network-diagnostics/network-check-target-gf2g6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:50.002776 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:50.002720    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f podName:9ec6705e-d32d-48c8-851f-0c353cf1463e nodeName:}" failed. No retries permitted until 2026-07-28 17:10:54.002702911 +0000 UTC m=+9.168338742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gj4f" (UniqueName: "kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f") pod "network-check-target-gf2g6" (UID: "9ec6705e-d32d-48c8-851f-0c353cf1463e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:50.357960 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:50.357926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:50.358129 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:50.358047    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:10:50.358129 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:50.358101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:50.358276 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:50.358253    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:10:51.358933 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:51.358380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:51.358933 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:51.358514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:10:52.358218 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:52.358178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:52.358408 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:52.358346    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:10:52.358748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:52.358702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:52.358877 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:52.358835    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:10:53.358670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:53.358054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:53.358670 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:53.358326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:10:53.936394 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:53.936355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:53.936585 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:53.936440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:53.936585 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:53.936530    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:53.936585 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:53.936553    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:53.936739 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:53.936608    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs podName:4fc84898-1712-4173-8e8a-3eb68293e69d nodeName:}" failed. No retries permitted until 2026-07-28 17:11:01.936586931 +0000 UTC m=+17.102222743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs") pod "network-metrics-daemon-rr4wq" (UID: "4fc84898-1712-4173-8e8a-3eb68293e69d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:53.936739 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:53.936630    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret podName:30185e23-4967-457e-836b-9a335ab75449 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:01.93661919 +0000 UTC m=+17.102255008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret") pod "global-pull-secret-syncer-9dlb9" (UID: "30185e23-4967-457e-836b-9a335ab75449") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:54.036884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:54.036846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:54.037085 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:54.037054    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:54.037085 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:54.037080    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:54.037085 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:54.037093    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9gj4f for pod openshift-network-diagnostics/network-check-target-gf2g6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:54.037330 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:54.037152    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f podName:9ec6705e-d32d-48c8-851f-0c353cf1463e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:02.037135043 +0000 UTC m=+17.202770869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gj4f" (UniqueName: "kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f") pod "network-check-target-gf2g6" (UID: "9ec6705e-d32d-48c8-851f-0c353cf1463e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:54.358377 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:54.358287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:54.358537 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:54.358424    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:10:54.358537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:54.358488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:54.358643 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:54.358616    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:10:55.359664 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:55.359208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:55.359664 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:55.359376    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:10:56.358589 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:56.358554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:56.358589 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:56.358577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:56.358795 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:56.358677    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:10:56.358884 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:56.358840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:10:57.358194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:57.358165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:57.358602 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:57.358269    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:10:58.358216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:58.358182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:10:58.358705 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:58.358182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:10:58.358705 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:58.358338    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:10:58.358705 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:58.358403    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:10:59.358009 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:10:59.357974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:10:59.358152 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:10:59.358121    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:00.358319 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:00.358270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:00.358778 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:00.358413    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:00.358778 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:00.358459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:00.358778 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:00.358569    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:01.357828 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:01.357791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:01.357993 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:01.357918    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:02.000331 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:02.000066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:02.000331 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.000257    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:02.000826 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:02.000334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:02.000826 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.000352    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret podName:30185e23-4967-457e-836b-9a335ab75449 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.000329551 +0000 UTC m=+33.165965396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret") pod "global-pull-secret-syncer-9dlb9" (UID: "30185e23-4967-457e-836b-9a335ab75449") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:02.000826 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.000419    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:02.000826 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.000474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs podName:4fc84898-1712-4173-8e8a-3eb68293e69d nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.000458225 +0000 UTC m=+33.166094048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs") pod "network-metrics-daemon-rr4wq" (UID: "4fc84898-1712-4173-8e8a-3eb68293e69d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:02.101111 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:02.101079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:02.101277 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.101207    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:11:02.101277 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.101229    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:11:02.101277 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.101239    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9gj4f for pod openshift-network-diagnostics/network-check-target-gf2g6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:02.101415 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.101290    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f podName:9ec6705e-d32d-48c8-851f-0c353cf1463e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.10127459 +0000 UTC m=+33.266910404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gj4f" (UniqueName: "kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f") pod "network-check-target-gf2g6" (UID: "9ec6705e-d32d-48c8-851f-0c353cf1463e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:02.357919 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:02.357887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:02.358149 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.358013    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:02.358149 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:02.358066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:02.358260 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:02.358164    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:03.357724 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:03.357698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:03.358173 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:03.357818    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:04.358140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:04.358100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:04.358563 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:04.358107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:04.358563 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:04.358244    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:04.358563 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:04.358340    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:05.359025 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:05.358989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:05.359500 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:05.359105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:06.358719 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.358526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:06.358859 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.358552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:06.358859 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:06.358807    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:06.358940 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:06.358867    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:06.495494 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.495437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b78kd" event={"ID":"112143bf-d024-4e4a-837a-c6c4b6421010","Type":"ContainerStarted","Data":"002dfe332abab7160a79dc9986b498492b88ebf7597e2adace157cad1e26198c"}
Jul 28 17:11:06.496903 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.496880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g757g" event={"ID":"c91bf743-8bcf-47c4-b0b5-90c088cbbd26","Type":"ContainerStarted","Data":"25e4c5e772eb02fd5a555694513f86a139893c0e3ac65698371441614f39957c"}
Jul 28 17:11:06.498211 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.498189    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1b81611-bb9f-4327-8b30-27fdf4699404" containerID="6fa80d2ab7d55e105e61964b7e73355eefbb7e0e7fd7b6086d33722177940ae9" exitCode=0
Jul 28 17:11:06.498487 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.498250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerDied","Data":"6fa80d2ab7d55e105e61964b7e73355eefbb7e0e7fd7b6086d33722177940ae9"}
Jul 28 17:11:06.500403 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.500383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"57682c76a28ab71a74683c7981fa62415acb7b88e6a3b0fb4fe13ad49461ef4b"}
Jul 28 17:11:06.500501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.500453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"a059b8d142a17de9fb00252f3a693ad40585c4351543aaf1670f68415da1ce78"}
Jul 28 17:11:06.500558 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.500505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"172cbec2890de1d8485adf8ead144dfe181bf8e505e2107d30c1d5fdebaa1f88"}
Jul 28 17:11:06.503867 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.503847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" event={"ID":"cfcf7e829014a4c1c014b92346f6e0a6","Type":"ContainerStarted","Data":"4dda845c94822a7e0fd2117f49599a80857b603b7d931b87b60b0d50acd7ea76"}
Jul 28 17:11:06.505121 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.505102    2578 generic.go:356] "Generic (PLEG): container finished" podID="e68a05610ec5fd1eced0e51b5c0733c5" containerID="2e9612edc2999969bc0575f947d99fb10a3921c801048cb785c7817b802ddaad" exitCode=0
Jul 28 17:11:06.505212 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.505172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"e68a05610ec5fd1eced0e51b5c0733c5","Type":"ContainerDied","Data":"2e9612edc2999969bc0575f947d99fb10a3921c801048cb785c7817b802ddaad"}
Jul 28 17:11:06.506430 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.506406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v7x9l" event={"ID":"f339ccd3-739e-4820-bbd6-1909f519163c","Type":"ContainerStarted","Data":"83b86ec261ed1d0ab95fe8f1f18cb038d5cef5a0a3a76529deb21762b657034a"}
Jul 28 17:11:06.507643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.507623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd" event={"ID":"e994bd0c-54f8-4539-a96e-59083cc6d258","Type":"ContainerStarted","Data":"53d6583f250c781125f471496da62cc10929412aa1883c9d9b37d7abe8f7543a"}
Jul 28 17:11:06.508806 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.508780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2d7hz" event={"ID":"4da92ddf-8155-41a3-bce0-e723bc0787c0","Type":"ContainerStarted","Data":"e00751585cc102beb1e7b51487c5deb2eb2b7d826cd47d661c88f1d883610a69"}
Jul 28 17:11:06.510159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.510140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kxnh" event={"ID":"1a2cb418-91da-4de5-a28e-ed1cf6ffea5f","Type":"ContainerStarted","Data":"02ac073ea6719c2d4b222b44a4239799afff2b32d56105fbdb42d63496f52f38"}
Jul 28 17:11:06.516347 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.516285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b78kd" podStartSLOduration=2.289008138 podStartE2EDuration="21.516273366s" totalImagesPullingTime="19.227265228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:06.515685344 +0000 UTC m=+21.681321179" watchObservedRunningTime="2026-07-28 17:11:06.516273366 +0000 UTC m=+21.681909199"
Jul 28 17:11:06.538348 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.538293    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v7x9l" podStartSLOduration=2.317186295 podStartE2EDuration="21.538279953s" totalImagesPullingTime="19.221093658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:06.538232818 +0000 UTC m=+21.703868652" watchObservedRunningTime="2026-07-28 17:11:06.538279953 +0000 UTC m=+21.703915787"
Jul 28 17:11:06.557879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.557848    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.ec2.internal" podStartSLOduration=20.557836975 podStartE2EDuration="20.557836975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:06.557706791 +0000 UTC m=+21.723342626" watchObservedRunningTime="2026-07-28 17:11:06.557836975 +0000 UTC m=+21.723472809"
Jul 28 17:11:06.573729 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.573693    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2kxnh" podStartSLOduration=2.34276129 podStartE2EDuration="21.573681711s" totalImagesPullingTime="19.230920421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:06.573255963 +0000 UTC m=+21.738891796" watchObservedRunningTime="2026-07-28 17:11:06.573681711 +0000 UTC m=+21.739317545"
Jul 28 17:11:06.620441 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.620403    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g757g" podStartSLOduration=2.336163797 podStartE2EDuration="21.620389127s" totalImagesPullingTime="19.28422533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:06.620086132 +0000 UTC m=+21.785721965" watchObservedRunningTime="2026-07-28 17:11:06.620389127 +0000 UTC m=+21.786024961"
Jul 28 17:11:06.637476 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.637442    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2d7hz" podStartSLOduration=2.372322418 podStartE2EDuration="21.637427452s" totalImagesPullingTime="19.265105034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:06.637221879 +0000 UTC m=+21.802857713" watchObservedRunningTime="2026-07-28 17:11:06.637427452 +0000 UTC m=+21.803063285"
Jul 28 17:11:06.646267 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.646246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:06.712029 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.712003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:06.809474 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.809250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:06.911622 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:06.911591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.358346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.358285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:07.358516 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:07.358435    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:07.513749 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.513670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcc6d" event={"ID":"d45de308-c906-41dd-91e0-b5d65ad0434e","Type":"ContainerStarted","Data":"54ba155368897f50e80ac3676fe620c9becd98899a5c19b0b4489711fab4bf1c"}
Jul 28 17:11:07.516745 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.516683    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"01b9841562e8f3d6e7152c7321cbd67573a338fd9baf699fcea90eda19499a78"}
Jul 28 17:11:07.516869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.516774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"d5cc1f0127ccfd9032439c76d5fb50f2fe7fa4f6545e16d9f4fb8d6e6768441c"}
Jul 28 17:11:07.516869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.516822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"87682745df251c0d1a903e36654ba3adfc024f7e03670233f5d0c78566d1892b"}
Jul 28 17:11:07.546558 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.546517    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fcc6d" podStartSLOduration=3.263359938 podStartE2EDuration="22.54650486s" totalImagesPullingTime="19.283144922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:07.530284084 +0000 UTC m=+22.695919951" watchObservedRunningTime="2026-07-28 17:11:07.54650486 +0000 UTC m=+22.712140694"
Jul 28 17:11:07.650244 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.650218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.728033 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.728007    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.760628 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.760263    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:11:07.907195 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.907153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.984058 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:07.984013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:08.249777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.249671    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:11:07.760279981Z","UUID":"d67b0ef3-81a3-446f-8035-2adbe9c9cf44","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:11:08.251554 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.251527    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:11:08.251554 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.251573    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:11:08.358213 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.358183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:08.358475 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.358183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:08.358475 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:08.358334    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:08.358475 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:08.358410    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:08.521830 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.521748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" event={"ID":"e68a05610ec5fd1eced0e51b5c0733c5","Type":"ContainerStarted","Data":"fcdb36d822d1c2d3a7911ac556835016588009036e60540472968bccd9413294"}
Jul 28 17:11:08.524047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.524022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd" event={"ID":"e994bd0c-54f8-4539-a96e-59083cc6d258","Type":"ContainerStarted","Data":"4b269815eb9b6bb097824f3600a6beae9a26f9064ba0e09adf1554104b48ddee"}
Jul 28 17:11:08.540055 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.539998    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.ec2.internal" podStartSLOduration=22.539979342 podStartE2EDuration="22.539979342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:08.539241831 +0000 UTC m=+23.704877666" watchObservedRunningTime="2026-07-28 17:11:08.539979342 +0000 UTC m=+23.705615177"
Jul 28 17:11:08.596143 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:08.596115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:09.358789 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:09.358755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:09.359068 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:09.358887    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:09.529094 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:09.529060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd" event={"ID":"e994bd0c-54f8-4539-a96e-59083cc6d258","Type":"ContainerStarted","Data":"8861dcd5c08513b00ffa294e1067cc6c8371b2e71e9f74d6d72cc111a9a1f733"}
Jul 28 17:11:09.532656 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:09.532623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"08a3f62e5f1bff8b918916308df17674cec43cea8c2c43a13d7a53c4aecda1bc"}
Jul 28 17:11:09.552533 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:09.552476    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f4cd" podStartSLOduration=3.261904639 podStartE2EDuration="24.552459287s" totalImagesPullingTime="21.290554648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:09.551802602 +0000 UTC m=+24.717438437" watchObservedRunningTime="2026-07-28 17:11:09.552459287 +0000 UTC m=+24.718095159"
Jul 28 17:11:10.320924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.320891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:10.357894 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.357859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:10.358075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.357862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:10.358075 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:10.358007    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:10.358075 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:10.358061    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:10.448890 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.448855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:10.559877 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.559841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:10.561748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.561717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:11:10.562388 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.562369    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:11:10.660149 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.660112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:10.890708 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:10.890673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.091649 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.091621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.206045 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.205792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.358036 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.358010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:11.358238 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:11.358208    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:11.541034 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.541009    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1b81611-bb9f-4327-8b30-27fdf4699404" containerID="7c09d0e9e0d2ba6cc9339225339c0b111e85a8eeab1ee357edc7e7f0221f5d23" exitCode=0
Jul 28 17:11:11.541147 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.541090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerDied","Data":"7c09d0e9e0d2ba6cc9339225339c0b111e85a8eeab1ee357edc7e7f0221f5d23"}
Jul 28 17:11:11.544591 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.544565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" event={"ID":"af516b76-8f7c-494c-9c49-d0b92829113d","Type":"ContainerStarted","Data":"aa93b527898d80762eafe9e4b846833e51ea23b6750c36b67dd0f63dda7acc5d"}
Jul 28 17:11:11.544923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.544893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:11:11.545453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.545436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2d7hz"
Jul 28 17:11:11.613942 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.613892    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l" podStartSLOduration=7.013776885 podStartE2EDuration="26.613879046s" totalImagesPullingTime="19.600102161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:11.613462281 +0000 UTC m=+26.779098115" watchObservedRunningTime="2026-07-28 17:11:11.613879046 +0000 UTC m=+26.779514880"
Jul 28 17:11:11.620416 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:11.620388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:12.032276 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.032242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:12.358503 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.358479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:12.358632 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.358480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:12.358632 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:12.358571    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:12.358702 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:12.358662    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:12.548173 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.548142    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1b81611-bb9f-4327-8b30-27fdf4699404" containerID="fc3b00fe300d4638f5210426dc4dcffadb694deba20acb0099c9950fb09d994e" exitCode=0
Jul 28 17:11:12.548359 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.548217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerDied","Data":"fc3b00fe300d4638f5210426dc4dcffadb694deba20acb0099c9950fb09d994e"}
Jul 28 17:11:12.548359 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.548275    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:11:12.548985 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.548845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:11:12.548985 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.548912    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:11:12.563810 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.563787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:11:12.563947 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.563917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:11:12.825858 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:12.825824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.055685 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.055649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.232036 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.232010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.358679 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.358653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:13.358811 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:13.358789    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:13.424040 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.424014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.552354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.552277    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1b81611-bb9f-4327-8b30-27fdf4699404" containerID="d6fa673313420ea44dbaeeee77843733c8212bf3eb1cf9f114c7b6b102f04b85" exitCode=0
Jul 28 17:11:13.552354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.552340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerDied","Data":"d6fa673313420ea44dbaeeee77843733c8212bf3eb1cf9f114c7b6b102f04b85"}
Jul 28 17:11:13.552531 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.552478    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:11:13.635875 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:13.635851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.028256 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.028231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.128034 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.128000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.358697 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.358667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:14.358891 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:14.358783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:14.358891 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.358851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:14.359012 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:14.358992    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:14.554701 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.554671    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:11:14.587716 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.587689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:11:14.616467 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.616393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.716912 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.716884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.819288 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.819254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.984910 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:14.984841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.088256 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:15.088223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.359632 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:15.359600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:15.359798 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:15.359700    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:15.421136 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:15.421104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.627449 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:15.627375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.729367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:15.729331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.021394 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:16.021314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.358494 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:16.358465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:16.358494 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:16.358483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:16.358960 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:16.358569    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:16.358960 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:16.358699    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:16.628821 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:16.628745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:17.030914 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:17.030821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:17.358225 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:17.358194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:17.358409 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:17.358312    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:17.425464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:17.425437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:17.603548 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:17.603520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:17.704759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:17.704692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:17.866701 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:17.866679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:17.970076 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:17.969921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:18.030270 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.030236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:18.030270 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.030289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:18.030516 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.030397    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:18.030516 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.030407    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:18.030516 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.030468    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret podName:30185e23-4967-457e-836b-9a335ab75449 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:50.030451928 +0000 UTC m=+65.196087739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret") pod "global-pull-secret-syncer-9dlb9" (UID: "30185e23-4967-457e-836b-9a335ab75449") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:18.030516 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.030485    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs podName:4fc84898-1712-4173-8e8a-3eb68293e69d nodeName:}" failed. No retries permitted until 2026-07-28 17:11:50.03047747 +0000 UTC m=+65.196113281 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs") pod "network-metrics-daemon-rr4wq" (UID: "4fc84898-1712-4173-8e8a-3eb68293e69d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:18.070740 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.070716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:18.130833 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.130801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:18.130974 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.130956    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:11:18.131017 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.130978    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:11:18.131017 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.130989    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9gj4f for pod openshift-network-diagnostics/network-check-target-gf2g6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:18.131087 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.131042    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f podName:9ec6705e-d32d-48c8-851f-0c353cf1463e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:50.13102054 +0000 UTC m=+65.296656351 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9gj4f" (UniqueName: "kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f") pod "network-check-target-gf2g6" (UID: "9ec6705e-d32d-48c8-851f-0c353cf1463e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:18.358220 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.358194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:18.358465 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.358286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:18.358465 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.358312    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:18.358465 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:18.358387    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:18.566436 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.566402    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1b81611-bb9f-4327-8b30-27fdf4699404" containerID="3ab7f3486cfb46653cb51f30ed3c3554eda57d255de03a0e2c3bc250772de694" exitCode=0
Jul 28 17:11:18.566796 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.566463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerDied","Data":"3ab7f3486cfb46653cb51f30ed3c3554eda57d255de03a0e2c3bc250772de694"}
Jul 28 17:11:18.629936 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:18.629913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.032758 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.032686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.174962 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.174935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.275310 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.275267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.358233 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.358212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:19.358378 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:19.358321    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:19.375582 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.375554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.476363 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.476333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.538594 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.538565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.571261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.571233    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1b81611-bb9f-4327-8b30-27fdf4699404" containerID="45a36a51e563d98ceaa3684725fb3ea956359fbfded5cc014e369cc4fd34a051" exitCode=0
Jul 28 17:11:19.571694 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.571291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerDied","Data":"45a36a51e563d98ceaa3684725fb3ea956359fbfded5cc014e369cc4fd34a051"}
Jul 28 17:11:19.619870 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.619801    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.681920 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.681877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.871012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:19.870948    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:20.068268 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.068234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:20.246914 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.246826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:20.310239 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.310216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:20.358441 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.358419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:20.358566 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:20.358505    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:20.358611 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.358561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:20.358670 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:20.358652    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:20.575974 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.575946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" event={"ID":"f1b81611-bb9f-4327-8b30-27fdf4699404","Type":"ContainerStarted","Data":"125f7fd7395a654f65394bca8a7e5e42e9d96c4f80a6c9c87f7b15c9140038db"}
Jul 28 17:11:20.604956 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.604913    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hxxn4" podStartSLOduration=5.950692631 podStartE2EDuration="35.604896913s" totalImagesPullingTime="29.654204282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:20.604749711 +0000 UTC m=+35.770385545" watchObservedRunningTime="2026-07-28 17:11:20.604896913 +0000 UTC m=+35.770532747"
Jul 28 17:11:20.624532 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.624510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:20.686121 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.686088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:20.814921 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:20.814900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.021049 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:21.020986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.120905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:21.120884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.358267 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:21.358242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:21.358443 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:21.358357    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:21.447537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:21.447515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.910643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:21.910607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:22.358291 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:22.358266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:22.358472 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:22.358370    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:22.358472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:22.358433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:22.358587 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:22.358552    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:23.357785 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:23.357758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:23.358132 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:23.357854    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:24.357848 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:24.357813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:24.358287 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:24.357817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:24.358287 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:24.357927    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:24.358287 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:24.357983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:25.358467 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:25.358435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:25.358821 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:25.358519    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:25.365257 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:25.365234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.461950 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:25.461903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.563355 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:25.563327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.944072 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:25.944043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.358180 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:26.358157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:26.358180 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:26.358171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:26.358377 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:26.358252    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:26.358414 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:26.358380    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:27.358435 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:27.358402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:27.358776 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:27.358500    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:28.358004 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:28.357977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:28.358170 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:28.357976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:28.358170 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:28.358093    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:28.358170 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:28.358128    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:29.358636 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:29.358610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:29.358970 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:29.358701    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:30.165451 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:30.165420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.241877 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:30.241848    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.346314 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:30.346278    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.357645 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:30.357624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:30.357762 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:30.357647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:30.357806 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:30.357752    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:30.357878 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:30.357861    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:30.445812 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:30.445737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.284782 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.284755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.352620 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.352596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.358538 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.358523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:31.358638 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:31.358616    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:31.496918 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.496884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gf2g6"]
Jul 28 17:11:31.497387 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.497005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:31.497387 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:31.497109    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:31.500664 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.500581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9dlb9"]
Jul 28 17:11:31.501381 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.501359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rr4wq"]
Jul 28 17:11:31.501590 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.501576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:31.501834 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:31.501809    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:31.598435 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.597800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:31.601138 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:31.601108    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:31.662333 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.662284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.759763 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:31.759691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.358187 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:33.357958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:33.358518 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:33.358010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:33.358518 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:33.358291    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:33.358518 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:33.358371    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:33.358518 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:33.358042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:33.358518 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:33.358442    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:35.358539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.358511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:35.358539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.358519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:35.358539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.358530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:35.359149 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:35.358620    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9dlb9" podUID="30185e23-4967-457e-836b-9a335ab75449"
Jul 28 17:11:35.359149 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:35.358746    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4wq" podUID="4fc84898-1712-4173-8e8a-3eb68293e69d"
Jul 28 17:11:35.359149 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:35.358803    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf2g6" podUID="9ec6705e-d32d-48c8-851f-0c353cf1463e"
Jul 28 17:11:35.618161 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.618092    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-66.ec2.internal" event="NodeReady"
Jul 28 17:11:35.618327 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.618226    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:11:35.676757 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.676734    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wvmd5"]
Jul 28 17:11:35.685062 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.685043    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hwb85"]
Jul 28 17:11:35.685172 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.685155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.688507 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.688485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:11:35.688605 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.688592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6mf7\""
Jul 28 17:11:35.688667 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.688592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:11:35.692765 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.692748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.692906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.692839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wvmd5"]
Jul 28 17:11:35.695860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.695839    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6z7jm\""
Jul 28 17:11:35.696011 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.695990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:11:35.696088 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.696034    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:11:35.697372 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.697354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:11:35.697464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.697402    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:11:35.701128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.701111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hwb85"]
Jul 28 17:11:35.703194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.703175    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jghvk"]
Jul 28 17:11:35.709469 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.709452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:35.713031 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.713013    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:11:35.713126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.713085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:11:35.713312 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.713282    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:11:35.713622 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.713610    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ncgbp\""
Jul 28 17:11:35.719896 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.719879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jghvk"]
Jul 28 17:11:35.762523 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1dacda5-83fd-4545-a3b1-be36741e44c7-data-volume\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.762636 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhw6p\" (UniqueName: \"kubernetes.io/projected/a1dacda5-83fd-4545-a3b1-be36741e44c7-kube-api-access-mhw6p\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.762636 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50e63a86-7a28-4b6e-93b9-8acefb97b80c-config-volume\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.762636 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50e63a86-7a28-4b6e-93b9-8acefb97b80c-metrics-tls\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.762817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74pgq\" (UniqueName: \"kubernetes.io/projected/86b63356-74c5-46ec-a2f6-66c82cf24df0-kube-api-access-74pgq\") pod \"ingress-canary-jghvk\" (UID: \"86b63356-74c5-46ec-a2f6-66c82cf24df0\") " pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:35.762817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmgh\" (UniqueName: \"kubernetes.io/projected/50e63a86-7a28-4b6e-93b9-8acefb97b80c-kube-api-access-jjmgh\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.762817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1dacda5-83fd-4545-a3b1-be36741e44c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.762817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b63356-74c5-46ec-a2f6-66c82cf24df0-cert\") pod \"ingress-canary-jghvk\" (UID: \"86b63356-74c5-46ec-a2f6-66c82cf24df0\") " pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:35.762817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e63a86-7a28-4b6e-93b9-8acefb97b80c-tmp-dir\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.763026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1dacda5-83fd-4545-a3b1-be36741e44c7-crio-socket\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.763026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.762882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1dacda5-83fd-4545-a3b1-be36741e44c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864004 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.863980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1dacda5-83fd-4545-a3b1-be36741e44c7-crio-socket\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864124 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1dacda5-83fd-4545-a3b1-be36741e44c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864124 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1dacda5-83fd-4545-a3b1-be36741e44c7-data-volume\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864124 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhw6p\" (UniqueName: \"kubernetes.io/projected/a1dacda5-83fd-4545-a3b1-be36741e44c7-kube-api-access-mhw6p\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864124 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1dacda5-83fd-4545-a3b1-be36741e44c7-crio-socket\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864352 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50e63a86-7a28-4b6e-93b9-8acefb97b80c-config-volume\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.864352 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50e63a86-7a28-4b6e-93b9-8acefb97b80c-metrics-tls\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.864352 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74pgq\" (UniqueName: \"kubernetes.io/projected/86b63356-74c5-46ec-a2f6-66c82cf24df0-kube-api-access-74pgq\") pod \"ingress-canary-jghvk\" (UID: \"86b63356-74c5-46ec-a2f6-66c82cf24df0\") " pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:35.864352 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmgh\" (UniqueName: \"kubernetes.io/projected/50e63a86-7a28-4b6e-93b9-8acefb97b80c-kube-api-access-jjmgh\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.864352 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1dacda5-83fd-4545-a3b1-be36741e44c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864579 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1dacda5-83fd-4545-a3b1-be36741e44c7-data-volume\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864579 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b63356-74c5-46ec-a2f6-66c82cf24df0-cert\") pod \"ingress-canary-jghvk\" (UID: \"86b63356-74c5-46ec-a2f6-66c82cf24df0\") " pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:35.864579 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e63a86-7a28-4b6e-93b9-8acefb97b80c-tmp-dir\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.864835 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1dacda5-83fd-4545-a3b1-be36741e44c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.864929 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.864843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50e63a86-7a28-4b6e-93b9-8acefb97b80c-config-volume\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.865628 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.865610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e63a86-7a28-4b6e-93b9-8acefb97b80c-tmp-dir\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.868065 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.868043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50e63a86-7a28-4b6e-93b9-8acefb97b80c-metrics-tls\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.868194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.868077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1dacda5-83fd-4545-a3b1-be36741e44c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.868313 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.868284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b63356-74c5-46ec-a2f6-66c82cf24df0-cert\") pod \"ingress-canary-jghvk\" (UID: \"86b63356-74c5-46ec-a2f6-66c82cf24df0\") " pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:35.869331 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.869290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:35.873862 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.873837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74pgq\" (UniqueName: \"kubernetes.io/projected/86b63356-74c5-46ec-a2f6-66c82cf24df0-kube-api-access-74pgq\") pod \"ingress-canary-jghvk\" (UID: \"86b63356-74c5-46ec-a2f6-66c82cf24df0\") " pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:35.874062 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.874047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmgh\" (UniqueName: \"kubernetes.io/projected/50e63a86-7a28-4b6e-93b9-8acefb97b80c-kube-api-access-jjmgh\") pod \"dns-default-wvmd5\" (UID: \"50e63a86-7a28-4b6e-93b9-8acefb97b80c\") " pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:35.874197 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.874180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhw6p\" (UniqueName: \"kubernetes.io/projected/a1dacda5-83fd-4545-a3b1-be36741e44c7-kube-api-access-mhw6p\") pod \"insights-runtime-extractor-hwb85\" (UID: \"a1dacda5-83fd-4545-a3b1-be36741e44c7\") " pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:35.952545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.952497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:35.995241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:35.995206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:36.002374 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.002359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hwb85"
Jul 28 17:11:36.017047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.017027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jghvk"
Jul 28 17:11:36.171732 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.171664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hwb85"]
Jul 28 17:11:36.175675 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.175627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wvmd5"]
Jul 28 17:11:36.175798 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:36.175714    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dacda5_83fd_4545_a3b1_be36741e44c7.slice/crio-510d6855cc4c9f8ed51a7d6260874db65c8194212608c7f1f1c0519654394df7 WatchSource:0}: Error finding container 510d6855cc4c9f8ed51a7d6260874db65c8194212608c7f1f1c0519654394df7: Status 404 returned error can't find the container with id 510d6855cc4c9f8ed51a7d6260874db65c8194212608c7f1f1c0519654394df7
Jul 28 17:11:36.176476 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.176444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jghvk"]
Jul 28 17:11:36.178416 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:36.178393    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b63356_74c5_46ec_a2f6_66c82cf24df0.slice/crio-6fcbf53e4461beb3b9a1accc39010803f5954efd71c7877bdd0a41a20512c748 WatchSource:0}: Error finding container 6fcbf53e4461beb3b9a1accc39010803f5954efd71c7877bdd0a41a20512c748: Status 404 returned error can't find the container with id 6fcbf53e4461beb3b9a1accc39010803f5954efd71c7877bdd0a41a20512c748
Jul 28 17:11:36.179195 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:36.179031    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e63a86_7a28_4b6e_93b9_8acefb97b80c.slice/crio-6a6e493175c5c1fd44c4f86a15cb2495843c11178163e28d07ea309ba6c81cd6 WatchSource:0}: Error finding container 6a6e493175c5c1fd44c4f86a15cb2495843c11178163e28d07ea309ba6c81cd6: Status 404 returned error can't find the container with id 6a6e493175c5c1fd44c4f86a15cb2495843c11178163e28d07ea309ba6c81cd6
Jul 28 17:11:36.477075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.476955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.613458 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.613408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwb85" event={"ID":"a1dacda5-83fd-4545-a3b1-be36741e44c7","Type":"ContainerStarted","Data":"510d6855cc4c9f8ed51a7d6260874db65c8194212608c7f1f1c0519654394df7"}
Jul 28 17:11:36.614878 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.614852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvmd5" event={"ID":"50e63a86-7a28-4b6e-93b9-8acefb97b80c","Type":"ContainerStarted","Data":"6a6e493175c5c1fd44c4f86a15cb2495843c11178163e28d07ea309ba6c81cd6"}
Jul 28 17:11:36.616056 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:36.616004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jghvk" event={"ID":"86b63356-74c5-46ec-a2f6-66c82cf24df0","Type":"ContainerStarted","Data":"6fcbf53e4461beb3b9a1accc39010803f5954efd71c7877bdd0a41a20512c748"}
Jul 28 17:11:37.358430 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.358398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:37.358614 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.358478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:37.358614 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.358582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:37.367545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.364500    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:37.367545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.364596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:37.367545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.365506    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-phqg2\""
Jul 28 17:11:37.367545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.365644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:37.367545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.366259    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:37.367545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.366259    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtwlq\""
Jul 28 17:11:37.670455 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.669738    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7898c74669-lj75b"]
Jul 28 17:11:37.674529 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.672574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.676043 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.676021    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:11:37.678014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.677436    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:11:37.678014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.677603    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hcgpx\""
Jul 28 17:11:37.678014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.677866    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:11:37.678014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.677900    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:11:37.678254 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.678103    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:11:37.678420 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.678370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:11:37.684060 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.680457    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:11:37.686101 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.686080    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:11:37.686261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.686100    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7898c74669-lj75b"]
Jul 28 17:11:37.781362 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.781270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-service-ca\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.781700 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.781671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sblq\" (UniqueName: \"kubernetes.io/projected/eef65afd-4517-4baa-936e-fc8dfb8eadef-kube-api-access-5sblq\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.781828 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.781739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-oauth-config\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.781828 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.781814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-trusted-ca-bundle\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.781943 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.781872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-serving-cert\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.781943 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.781895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-config\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.781943 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.781917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-oauth-serving-cert\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.882612 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.882576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-serving-cert\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.882798 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.882618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-config\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.882798 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.882652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-oauth-serving-cert\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.882798 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.882690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-service-ca\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.882798 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.882717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sblq\" (UniqueName: \"kubernetes.io/projected/eef65afd-4517-4baa-936e-fc8dfb8eadef-kube-api-access-5sblq\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.882798 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.882765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-oauth-config\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.883046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.882832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-trusted-ca-bundle\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.883823 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.883428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-config\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.884000 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.883977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-oauth-serving-cert\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.884260 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.884210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-service-ca\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.884570 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.884527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-trusted-ca-bundle\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.886964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.886939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-oauth-config\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.887224 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.887199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-serving-cert\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.893998 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.893946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sblq\" (UniqueName: \"kubernetes.io/projected/eef65afd-4517-4baa-936e-fc8dfb8eadef-kube-api-access-5sblq\") pod \"console-7898c74669-lj75b\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") " pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:37.989088 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:37.989002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:38.611957 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:38.611923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:38.669362 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:38.669325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.306209 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:39.306177    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.372339 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:39.372274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:40.080858 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:40.080830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:40.146231 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:40.146191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:41.734121 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:41.734093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7898c74669-lj75b"]
Jul 28 17:11:41.738750 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:41.738717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef65afd_4517_4baa_936e_fc8dfb8eadef.slice/crio-d19d4c1561fed55353968ea9f31fc608287059c22438a325b662330670474bbb WatchSource:0}: Error finding container d19d4c1561fed55353968ea9f31fc608287059c22438a325b662330670474bbb: Status 404 returned error can't find the container with id d19d4c1561fed55353968ea9f31fc608287059c22438a325b662330670474bbb
Jul 28 17:11:42.145880 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.145837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.245363 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.245260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.599520 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.599489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.616716 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.616692    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xcg59"]
Jul 28 17:11:42.631707 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.631667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xcg59"]
Jul 28 17:11:42.631870 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.631844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.638057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.637959    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:11:42.638057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.637979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5g9f5\""
Jul 28 17:11:42.638310 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.638151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:11:42.638829 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.638766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:11:42.639054 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.639033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:11:42.639138 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.639081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:11:42.639138 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.639119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvmd5" event={"ID":"50e63a86-7a28-4b6e-93b9-8acefb97b80c","Type":"ContainerStarted","Data":"3645e4b80a7979c9cae59ffec8815e7596c4b057aea151a96b47e4a33aa5ff9c"}
Jul 28 17:11:42.639242 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.639167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvmd5" event={"ID":"50e63a86-7a28-4b6e-93b9-8acefb97b80c","Type":"ContainerStarted","Data":"3fba6e18af337d3df30b8a63a88261c73cdc6151e5ea1a5ce5061bb05d7168de"}
Jul 28 17:11:42.639319 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.639284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:42.641134 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.641107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jghvk" event={"ID":"86b63356-74c5-46ec-a2f6-66c82cf24df0","Type":"ContainerStarted","Data":"6acda63533cc009c4b88816a6e821d5bf3c4c97e4d5aedc5a6346bbe5931e9b5"}
Jul 28 17:11:42.644595 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.644506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwb85" event={"ID":"a1dacda5-83fd-4545-a3b1-be36741e44c7","Type":"ContainerStarted","Data":"4a6139ac6d4b8ed75f29bba565e3db4f7336385af29c0d1aba9b5dd8f9bcc99c"}
Jul 28 17:11:42.646017 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.645993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7898c74669-lj75b" event={"ID":"eef65afd-4517-4baa-936e-fc8dfb8eadef","Type":"ContainerStarted","Data":"d19d4c1561fed55353968ea9f31fc608287059c22438a325b662330670474bbb"}
Jul 28 17:11:42.677677 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.677629    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wvmd5" podStartSLOduration=2.255106282 podStartE2EDuration="7.677615671s" totalImagesPullingTime="5.422509389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:42.677336614 +0000 UTC m=+57.842972450" watchObservedRunningTime="2026-07-28 17:11:42.677615671 +0000 UTC m=+57.843251504"
Jul 28 17:11:42.692955 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.692918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.725157 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.725117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.725288 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.725190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzf6h\" (UniqueName: \"kubernetes.io/projected/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-kube-api-access-gzf6h\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.725385 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.725275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.725385 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.725357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.794769 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.794735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.826326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.826280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.826472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.826379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzf6h\" (UniqueName: \"kubernetes.io/projected/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-kube-api-access-gzf6h\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.826472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.826427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.826472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.826467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.827214 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.827166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.830212 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.830187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.831282 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.831260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.845817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.845770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzf6h\" (UniqueName: \"kubernetes.io/projected/60f8208a-8e4e-4ac6-8f55-d3bbd7984f82-kube-api-access-gzf6h\") pod \"prometheus-operator-84d69f794d-xcg59\" (UID: \"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:42.917705 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.917557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.944670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:42.944642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59"
Jul 28 17:11:43.014235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:43.014197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:43.119587 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:43.119519    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jghvk" podStartSLOduration=2.694727671 podStartE2EDuration="8.11949587s" totalImagesPullingTime="5.424768199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:42.699807327 +0000 UTC m=+57.865443163" watchObservedRunningTime="2026-07-28 17:11:43.11949587 +0000 UTC m=+58.285131705"
Jul 28 17:11:43.120053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:43.120028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xcg59"]
Jul 28 17:11:43.127236 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:43.127201    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f8208a_8e4e_4ac6_8f55_d3bbd7984f82.slice/crio-5426af2baf11d8d1f6e4174b35ded00fd1382208ba8c884df225ea29ae0f40d4 WatchSource:0}: Error finding container 5426af2baf11d8d1f6e4174b35ded00fd1382208ba8c884df225ea29ae0f40d4: Status 404 returned error can't find the container with id 5426af2baf11d8d1f6e4174b35ded00fd1382208ba8c884df225ea29ae0f40d4
Jul 28 17:11:43.651773 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:43.651695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59" event={"ID":"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82","Type":"ContainerStarted","Data":"5426af2baf11d8d1f6e4174b35ded00fd1382208ba8c884df225ea29ae0f40d4"}
Jul 28 17:11:45.569736 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:45.569708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qdr2l"
Jul 28 17:11:45.756577 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:45.756528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:45.834729 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:45.834686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:45.915967 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:45.915935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:47.116879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.116849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:47.294028 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.293928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:47.397219 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.397189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:47.664456 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.664425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwb85" event={"ID":"a1dacda5-83fd-4545-a3b1-be36741e44c7","Type":"ContainerStarted","Data":"93d89dd7608fdd05e9591ae97ee4bde487bba79a70eb3ffe747ea9fc4865fa1d"}
Jul 28 17:11:47.665854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.665828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7898c74669-lj75b" event={"ID":"eef65afd-4517-4baa-936e-fc8dfb8eadef","Type":"ContainerStarted","Data":"542e0fdea953d8ab962455697a0b9c7af59619e52335450d92054a38fdf8a1a3"}
Jul 28 17:11:47.667482 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.667458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59" event={"ID":"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82","Type":"ContainerStarted","Data":"6c43a2ff00157bd302fdb75f1c4d330ab80dccfd99b1ad4d1f09f27eca9b0bf8"}
Jul 28 17:11:47.667604 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.667499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59" event={"ID":"60f8208a-8e4e-4ac6-8f55-d3bbd7984f82","Type":"ContainerStarted","Data":"f0ef8695c539db6db656b3e917f330f10ac4631df7040826c5157d1ceba6db7f"}
Jul 28 17:11:47.692930 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.692885    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7898c74669-lj75b" podStartSLOduration=5.447961508 podStartE2EDuration="10.6928729s" totalImagesPullingTime="5.244911392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:47.690876697 +0000 UTC m=+62.856512542" watchObservedRunningTime="2026-07-28 17:11:47.6928729 +0000 UTC m=+62.858508734"
Jul 28 17:11:47.713252 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.713212    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-xcg59" podStartSLOduration=1.776631222 podStartE2EDuration="5.713200737s" totalImagesPullingTime="3.936569515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:47.712283394 +0000 UTC m=+62.877919239" watchObservedRunningTime="2026-07-28 17:11:47.713200737 +0000 UTC m=+62.878836571"
Jul 28 17:11:47.739468 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.739442    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:47.869897 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.869862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:47.989783 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.989702    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:47.989919 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.989784    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:47.996247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:47.996223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:48.118002 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:48.117967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:48.221132 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:48.221102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:48.675841 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:48.675813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:11:48.867273 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:48.867234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:48.947416 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:48.947320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.132453 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:49.132402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.224676 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:49.224595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.717238 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:49.717205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.870493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:49.870459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.999130 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:49.999047    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:50.037583 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.037557    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8jbfr"]
Jul 28 17:11:50.088042 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.088002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:50.090565 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.090541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:50.090714 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.090587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:50.094063 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.094044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:50.094157 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.094047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:50.103252 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.103233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc84898-1712-4173-8e8a-3eb68293e69d-metrics-certs\") pod \"network-metrics-daemon-rr4wq\" (UID: \"4fc84898-1712-4173-8e8a-3eb68293e69d\") " pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:50.103363 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.103313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30185e23-4967-457e-836b-9a335ab75449-original-pull-secret\") pod \"global-pull-secret-syncer-9dlb9\" (UID: \"30185e23-4967-457e-836b-9a335ab75449\") " pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:50.113140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.113125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.116522 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.116368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:11:50.116522 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.116386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:11:50.116723 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.116704    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zrj4d\""
Jul 28 17:11:50.116782 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.116720    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:11:50.191023 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.190983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-tls\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-sys\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-wtmp\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/966c33cf-d8f8-49f3-aea9-2cea30006c68-metrics-client-ca\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-root\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-textfile\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.191501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.191437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24cm\" (UniqueName: \"kubernetes.io/projected/966c33cf-d8f8-49f3-aea9-2cea30006c68-kube-api-access-s24cm\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.192161 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.192132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:50.194753 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.194732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:50.205326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.205289    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:50.215811 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.215788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gj4f\" (UniqueName: \"kubernetes.io/projected/9ec6705e-d32d-48c8-851f-0c353cf1463e-kube-api-access-9gj4f\") pod \"network-check-target-gf2g6\" (UID: \"9ec6705e-d32d-48c8-851f-0c353cf1463e\") " pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:50.270455 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.270377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:50.279390 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.279363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtwlq\""
Jul 28 17:11:50.285583 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.285561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9dlb9"
Jul 28 17:11:50.286612 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.286593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4wq"
Jul 28 17:11:50.291987 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.291966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/966c33cf-d8f8-49f3-aea9-2cea30006c68-metrics-client-ca\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292103 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-root\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292103 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-textfile\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292103 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24cm\" (UniqueName: \"kubernetes.io/projected/966c33cf-d8f8-49f3-aea9-2cea30006c68-kube-api-access-s24cm\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-root\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-tls\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-sys\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-wtmp\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292550 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-textfile\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292550 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-sys\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292696 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-wtmp\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292809 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/966c33cf-d8f8-49f3-aea9-2cea30006c68-metrics-client-ca\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.292809 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.292779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.294696 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.294673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-tls\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.294819 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.294792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/966c33cf-d8f8-49f3-aea9-2cea30006c68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.295594 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.295573    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-phqg2\""
Jul 28 17:11:50.302572 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.302554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:50.302883 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.302868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24cm\" (UniqueName: \"kubernetes.io/projected/966c33cf-d8f8-49f3-aea9-2cea30006c68-kube-api-access-s24cm\") pod \"node-exporter-8jbfr\" (UID: \"966c33cf-d8f8-49f3-aea9-2cea30006c68\") " pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.339546 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.339512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:50.424623 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.424588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jbfr"
Jul 28 17:11:50.498443 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.498408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:50.719453 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:50.719417    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966c33cf_d8f8_49f3_aea9_2cea30006c68.slice/crio-65d5e6da101bdac700af77041a020524f3bc1b44be0e875fe9cf60b75697bcf8 WatchSource:0}: Error finding container 65d5e6da101bdac700af77041a020524f3bc1b44be0e875fe9cf60b75697bcf8: Status 404 returned error can't find the container with id 65d5e6da101bdac700af77041a020524f3bc1b44be0e875fe9cf60b75697bcf8
Jul 28 17:11:50.859945 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.859875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gf2g6"]
Jul 28 17:11:50.861901 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:50.861875    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec6705e_d32d_48c8_851f_0c353cf1463e.slice/crio-c2e4b9130989164211ca362ceef8e67e7361432353498431cebd93d765c3f7fd WatchSource:0}: Error finding container c2e4b9130989164211ca362ceef8e67e7361432353498431cebd93d765c3f7fd: Status 404 returned error can't find the container with id c2e4b9130989164211ca362ceef8e67e7361432353498431cebd93d765c3f7fd
Jul 28 17:11:50.971812 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:50.971731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.076466 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.076434    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:51.080075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.080052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.081018 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.080996    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9dlb9"]
Jul 28 17:11:51.083928 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:51.083869    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30185e23_4967_457e_836b_9a335ab75449.slice/crio-d18d19efd9325f6d6bfb891faa9a5042b6a693292c32303060c3ba0c06b8c019 WatchSource:0}: Error finding container d18d19efd9325f6d6bfb891faa9a5042b6a693292c32303060c3ba0c06b8c019: Status 404 returned error can't find the container with id d18d19efd9325f6d6bfb891faa9a5042b6a693292c32303060c3ba0c06b8c019
Jul 28 17:11:51.084075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.083967    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mfpvw\""
Jul 28 17:11:51.084075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.083982    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:11:51.084075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.083990    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:11:51.084075 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.084064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:11:51.084270 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.084205    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:11:51.084530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.084483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:11:51.084667 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.084562    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:11:51.084667 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.084638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:11:51.084883 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.084862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:11:51.084983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.084965    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:11:51.086666 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.086643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.087053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.087032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rr4wq"]
Jul 28 17:11:51.089992 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:51.089972    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc84898_1712_4173_8e8a_3eb68293e69d.slice/crio-cf0fee6cf075d75771dedac971938cfc2038a95613dab81eec278a3e6b1bd329 WatchSource:0}: Error finding container cf0fee6cf075d75771dedac971938cfc2038a95613dab81eec278a3e6b1bd329: Status 404 returned error can't find the container with id cf0fee6cf075d75771dedac971938cfc2038a95613dab81eec278a3e6b1bd329
Jul 28 17:11:51.093940 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.093848    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:51.201698 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-out\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-web-config\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.201998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9ct\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-kube-api-access-tj9ct\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.202024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.202094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202541 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.202142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.202541 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.202164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.285013 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.284933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.303044 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303190 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303190 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303322 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303322 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-web-config\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9ct\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-kube-api-access-tj9ct\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303823 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-out\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.303823 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.303645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.304510 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:11:51.304050    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle podName:53408a8c-ffb1-41ce-82fb-7ed0cd3312f6 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:51.804027805 +0000 UTC m=+66.969663636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6") : configmap references non-existent config key: ca-bundle.crt
Jul 28 17:11:51.305858 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.305361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.308842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.308548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.308842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.308599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.308842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.308698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.308842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.308767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.308842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.308791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-out\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.309156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.309052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.309156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.309126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.309309 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.309163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-web-config\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.313020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.310249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.314887 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.314863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9ct\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-kube-api-access-tj9ct\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.385145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.385111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.469148 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.469110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.682133 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.682102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbfr" event={"ID":"966c33cf-d8f8-49f3-aea9-2cea30006c68","Type":"ContainerStarted","Data":"65d5e6da101bdac700af77041a020524f3bc1b44be0e875fe9cf60b75697bcf8"}
Jul 28 17:11:51.683515 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.683483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gf2g6" event={"ID":"9ec6705e-d32d-48c8-851f-0c353cf1463e","Type":"ContainerStarted","Data":"c2e4b9130989164211ca362ceef8e67e7361432353498431cebd93d765c3f7fd"}
Jul 28 17:11:51.686816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.686781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwb85" event={"ID":"a1dacda5-83fd-4545-a3b1-be36741e44c7","Type":"ContainerStarted","Data":"3552a7e0b892b82276cab7024cfc5d67d7ee4b857404af798d5fa804fa144995"}
Jul 28 17:11:51.688761 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.688732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9dlb9" event={"ID":"30185e23-4967-457e-836b-9a335ab75449","Type":"ContainerStarted","Data":"d18d19efd9325f6d6bfb891faa9a5042b6a693292c32303060c3ba0c06b8c019"}
Jul 28 17:11:51.690068 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.690047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rr4wq" event={"ID":"4fc84898-1712-4173-8e8a-3eb68293e69d","Type":"ContainerStarted","Data":"cf0fee6cf075d75771dedac971938cfc2038a95613dab81eec278a3e6b1bd329"}
Jul 28 17:11:51.717845 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.716673    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hwb85" podStartSLOduration=2.299540681 podStartE2EDuration="16.716653423s" totalImagesPullingTime="14.417112742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:51.716345494 +0000 UTC m=+66.881981355" watchObservedRunningTime="2026-07-28 17:11:51.716653423 +0000 UTC m=+66.882289262"
Jul 28 17:11:51.808475 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.808441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.810424 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.810396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:51.991052 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:51.990967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:52.245661 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.245502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:52.348846 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:52.348807    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53408a8c_ffb1_41ce_82fb_7ed0cd3312f6.slice/crio-bd896aec85a78d75e0f61d591e4dbe76fbacca765759ee059031c307ef637cbd WatchSource:0}: Error finding container bd896aec85a78d75e0f61d591e4dbe76fbacca765759ee059031c307ef637cbd: Status 404 returned error can't find the container with id bd896aec85a78d75e0f61d591e4dbe76fbacca765759ee059031c307ef637cbd
Jul 28 17:11:52.653937 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.653911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wvmd5"
Jul 28 17:11:52.699099 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.699030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerStarted","Data":"bd896aec85a78d75e0f61d591e4dbe76fbacca765759ee059031c307ef637cbd"}
Jul 28 17:11:52.702490 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.701958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rr4wq" event={"ID":"4fc84898-1712-4173-8e8a-3eb68293e69d","Type":"ContainerStarted","Data":"0f4309353711097f77c60a3b88ade7a4b454a2497b3a0665043a3fba09ea8ce3"}
Jul 28 17:11:52.702490 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.702009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rr4wq" event={"ID":"4fc84898-1712-4173-8e8a-3eb68293e69d","Type":"ContainerStarted","Data":"e062d84aabd007e95cae904c934867abfb4cd1523654dc3f67781e5360e8cf14"}
Jul 28 17:11:52.705596 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.705055    2578 generic.go:356] "Generic (PLEG): container finished" podID="966c33cf-d8f8-49f3-aea9-2cea30006c68" containerID="be8c4ab8e55e2e502ada945ed52c56c88132c743ae402a2de4760d43a03f50a1" exitCode=0
Jul 28 17:11:52.705596 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.705324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbfr" event={"ID":"966c33cf-d8f8-49f3-aea9-2cea30006c68","Type":"ContainerDied","Data":"be8c4ab8e55e2e502ada945ed52c56c88132c743ae402a2de4760d43a03f50a1"}
Jul 28 17:11:52.722806 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.722614    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rr4wq" podStartSLOduration=66.422676441 podStartE2EDuration="1m7.722596224s" totalImagesPullingTime="1.299919783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:52.720977271 +0000 UTC m=+67.886613107" watchObservedRunningTime="2026-07-28 17:11:52.722596224 +0000 UTC m=+67.888232058"
Jul 28 17:11:52.827063 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.827000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:52.927942 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:52.927902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.030349 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.030216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.244044 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.244004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.439936 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.439894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.680108 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.680069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.716553 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.716440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbfr" event={"ID":"966c33cf-d8f8-49f3-aea9-2cea30006c68","Type":"ContainerStarted","Data":"3718c23a4f0d1b8607f621f8f3277110b60baeffa454c345b142d5cb40f9075b"}
Jul 28 17:11:53.716553 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.716529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jbfr" event={"ID":"966c33cf-d8f8-49f3-aea9-2cea30006c68","Type":"ContainerStarted","Data":"93d5ff1add3de16c7fa1b2980ac6559ef57dc081dea8909b5f78fd78da633c68"}
Jul 28 17:11:53.745718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.745665    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8jbfr" podStartSLOduration=2.728967944 podStartE2EDuration="3.745649313s" totalImagesPullingTime="926.710254ms" totalInitContainerRuntime="89.971115ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:53.743450676 +0000 UTC m=+68.909086514" watchObservedRunningTime="2026-07-28 17:11:53.745649313 +0000 UTC m=+68.911285145"
Jul 28 17:11:53.781323 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.781270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.840529 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:53.840497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.184418 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.184381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.279796 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.279753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.287735 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.287700    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f48c497b6-7hqb5"]
Jul 28 17:11:54.290906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.290883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.294020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.293996    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:11:54.295685 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.295559    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cq6rd\""
Jul 28 17:11:54.295787 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.295712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:11:54.295867 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.295842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:11:54.295992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.295981    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c00buk1npopc7\""
Jul 28 17:11:54.296087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.296067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:11:54.301472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.301451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f48c497b6-7hqb5"]
Jul 28 17:11:54.432485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.432453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-audit-log\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.432619 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.432516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-secret-metrics-server-tls\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.432619 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.432538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-secret-metrics-server-client-certs\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.432619 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.432568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqwd9\" (UniqueName: \"kubernetes.io/projected/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-kube-api-access-fqwd9\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.432619 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.432591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-metrics-server-audit-profiles\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.432619 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.432607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.432792 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.432628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-client-ca-bundle\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.533443 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-secret-metrics-server-tls\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.533443 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-secret-metrics-server-client-certs\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.533443 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqwd9\" (UniqueName: \"kubernetes.io/projected/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-kube-api-access-fqwd9\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.533443 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-metrics-server-audit-profiles\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.534014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.534014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-client-ca-bundle\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.534014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-audit-log\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.534014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.533904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-audit-log\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.534425 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.534397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.534595 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.534574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-metrics-server-audit-profiles\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.536275 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.536250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-secret-metrics-server-client-certs\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.536415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.536399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-secret-metrics-server-tls\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.536479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.536442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-client-ca-bundle\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.542990 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.542951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqwd9\" (UniqueName: \"kubernetes.io/projected/f64e18a6-bf35-4fd4-ad2b-69151ec224b4-kube-api-access-fqwd9\") pod \"metrics-server-7f48c497b6-7hqb5\" (UID: \"f64e18a6-bf35-4fd4-ad2b-69151ec224b4\") " pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.602853 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.602824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:11:54.746331 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.746261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.845583 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.845553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.948808 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:54.948761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:55.044538 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:55.044493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.064354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.064325    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f48c497b6-7hqb5"]
Jul 28 17:11:56.336862 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.332327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.336862 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.332603    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:11:56.340562 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.340538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.345679 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.345661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:11:56.345823 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.345805    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:11:56.345901 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.345817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:11:56.346500 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.346472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:11:56.346500 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.346485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:11:56.346647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.346498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:11:56.346647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.346478    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:11:56.346647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.346585    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sg5kw\""
Jul 28 17:11:56.346872 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.346856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:11:56.347499 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.347475    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:11:56.347641 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.347622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:11:56.347728 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.347654    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:11:56.347777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.347751    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:11:56.349038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.348114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:11:56.349038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.348143    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9vkf9f9gt4dg5\""
Jul 28 17:11:56.350427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.350408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:11:56.360441 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.360416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:11:56.448795 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.448760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.448951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.448805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.448951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.448834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.448951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.448906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449080 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.448977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449080 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449080 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449080 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449080 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449281 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449281 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-config\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449281 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449281 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449281 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4wz\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-kube-api-access-2z4wz\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.449470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.449453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550097 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550097 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550349 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-config\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550349 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550349 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550504 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550504 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550504 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4wz\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-kube-api-access-2z4wz\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550504 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550504 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.550741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.551105 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.551105 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.550963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.552052 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.551830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.552052 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.552034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.552634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.552608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.552848 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.552824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.553488 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.553286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.553488 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.553356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.553488 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.553396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.554407 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.554381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.555670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.554843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.555670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.555150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.555670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.555267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-config\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.555670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.555392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.555670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.555610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.555670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.555656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.555988 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.555959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.556448 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.556424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.556552 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.556485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.557799 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.557047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.557799 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.557061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.565736 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.565718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4wz\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-kube-api-access-2z4wz\") pod \"prometheus-k8s-0\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.652119 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.651961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:11:56.733979 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.733907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gf2g6" event={"ID":"9ec6705e-d32d-48c8-851f-0c353cf1463e","Type":"ContainerStarted","Data":"02cbd68d0ba8eed358976b8d3ff71bb229b9e3e79f07648a5577ca3d15321875"}
Jul 28 17:11:56.734187 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.734051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:11:56.736008 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.735968    2578 generic.go:356] "Generic (PLEG): container finished" podID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerID="28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2" exitCode=0
Jul 28 17:11:56.736137 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.736052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2"}
Jul 28 17:11:56.738294 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.738115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5" event={"ID":"f64e18a6-bf35-4fd4-ad2b-69151ec224b4","Type":"ContainerStarted","Data":"578d84bffbdff30305037f6f86fb4bcaaa2123e01b6db041eb0b884afee9d340"}
Jul 28 17:11:56.743027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.740230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9dlb9" event={"ID":"30185e23-4967-457e-836b-9a335ab75449","Type":"ContainerStarted","Data":"20a88ab63f7c077146152f7cd9de309e8c8b731df1d586baf4e4950c83793271"}
Jul 28 17:11:56.824043 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.823978    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gf2g6" podStartSLOduration=66.776276993 podStartE2EDuration="1m11.823955872s" totalImagesPullingTime="5.047678879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:56.777601029 +0000 UTC m=+71.943236864" watchObservedRunningTime="2026-07-28 17:11:56.823955872 +0000 UTC m=+71.989591703"
Jul 28 17:11:56.848155 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.848055    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:11:56.853829 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:11:56.853797    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebf3fea_6e9c_4459_b332_03ad67904f88.slice/crio-d230ef9d3b38f9ddf3dde0e650b208f55185b21b36f3ad37ae2d12a46152a6d7 WatchSource:0}: Error finding container d230ef9d3b38f9ddf3dde0e650b208f55185b21b36f3ad37ae2d12a46152a6d7: Status 404 returned error can't find the container with id d230ef9d3b38f9ddf3dde0e650b208f55185b21b36f3ad37ae2d12a46152a6d7
Jul 28 17:11:56.865640 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:56.865599    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9dlb9" podStartSLOduration=67.035839646 podStartE2EDuration="1m11.865580149s" totalImagesPullingTime="4.829740503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:56.864322955 +0000 UTC m=+72.029958789" watchObservedRunningTime="2026-07-28 17:11:56.865580149 +0000 UTC m=+72.031216014"
Jul 28 17:11:57.747367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:57.747136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5" event={"ID":"f64e18a6-bf35-4fd4-ad2b-69151ec224b4","Type":"ContainerStarted","Data":"d1a05cfaa2f3a3c49f1172f279848fab8997473c2786f5172ce595a12f7a56de"}
Jul 28 17:11:57.748905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:57.748873    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314" exitCode=0
Jul 28 17:11:57.749046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:57.748949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"}
Jul 28 17:11:57.749046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:57.749027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerStarted","Data":"d230ef9d3b38f9ddf3dde0e650b208f55185b21b36f3ad37ae2d12a46152a6d7"}
Jul 28 17:11:57.768269 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:57.768227    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5" podStartSLOduration=2.364082847 podStartE2EDuration="3.76821212s" totalImagesPullingTime="1.404129273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:57.767463195 +0000 UTC m=+72.933099041" watchObservedRunningTime="2026-07-28 17:11:57.76821212 +0000 UTC m=+72.933847953"
Jul 28 17:11:58.757077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:58.757042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerStarted","Data":"b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28"}
Jul 28 17:11:58.757471 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:58.757137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerStarted","Data":"5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5"}
Jul 28 17:11:59.764681 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:59.764649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerStarted","Data":"0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e"}
Jul 28 17:11:59.765110 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:59.764716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerStarted","Data":"729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309"}
Jul 28 17:11:59.765110 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:11:59.764751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerStarted","Data":"bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902"}
Jul 28 17:12:00.572584 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:00.572542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:00.672769 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:00.672738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.774617 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:01.774584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerStarted","Data":"00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0"}
Jul 28 17:12:01.776720 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:01.776697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerStarted","Data":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"}
Jul 28 17:12:01.776816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:01.776749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerStarted","Data":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"}
Jul 28 17:12:01.810943 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:01.810896    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.157579243 podStartE2EDuration="10.810881047s" totalImagesPullingTime="7.619933298s" totalInitContainerRuntime="33.368506ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:01.808770179 +0000 UTC m=+76.974406030" watchObservedRunningTime="2026-07-28 17:12:01.810881047 +0000 UTC m=+76.976516880"
Jul 28 17:12:03.704285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:03.704251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:03.788935 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:03.788906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerStarted","Data":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"}
Jul 28 17:12:03.789060 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:03.788962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerStarted","Data":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"}
Jul 28 17:12:03.789060 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:03.788990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerStarted","Data":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"}
Jul 28 17:12:03.789060 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:03.789013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerStarted","Data":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"}
Jul 28 17:12:03.804869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:03.804842    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:03.826906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:03.826852    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.746261036 podStartE2EDuration="7.826832904s" totalImagesPullingTime="5.063356046s" totalInitContainerRuntime="17.215822ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:03.824355701 +0000 UTC m=+78.989991534" watchObservedRunningTime="2026-07-28 17:12:03.826832904 +0000 UTC m=+78.992468738"
Jul 28 17:12:06.200505 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:06.200459    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7898c74669-lj75b"]
Jul 28 17:12:06.298069 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:06.298043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:06.399757 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:06.399720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:06.500482 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:06.500405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:06.652354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:06.652324    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:07.069009 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:07.068904    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jghvk_86b63356-74c5-46ec-a2f6-66c82cf24df0/serve-healthcheck-canary/0.log"
Jul 28 17:12:13.092742 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:13.092676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:13.191543 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:13.191509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:13.290203 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:13.290171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:14.603244 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:14.603212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:12:14.603642 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:14.603271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:12:20.277058 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:20.276952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:20.376483 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:20.376444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:24.374783 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:24.374748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:24.431407 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:24.431373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:24.490747 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:24.490714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:27.751280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:27.751244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gf2g6"
Jul 28 17:12:31.219261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:31.219217    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7898c74669-lj75b" podUID="eef65afd-4517-4baa-936e-fc8dfb8eadef" containerName="console" containerID="cri-o://542e0fdea953d8ab962455697a0b9c7af59619e52335450d92054a38fdf8a1a3" gracePeriod=15
Jul 28 17:12:31.895421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:31.895379    2578 generic.go:356] "Generic (PLEG): container finished" podID="eef65afd-4517-4baa-936e-fc8dfb8eadef" containerID="542e0fdea953d8ab962455697a0b9c7af59619e52335450d92054a38fdf8a1a3" exitCode=0
Jul 28 17:12:31.895557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:31.895417    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7898c74669-lj75b" event={"ID":"eef65afd-4517-4baa-936e-fc8dfb8eadef","Type":"ContainerDied","Data":"542e0fdea953d8ab962455697a0b9c7af59619e52335450d92054a38fdf8a1a3"}
Jul 28 17:12:31.993776 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:31.993749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:12:32.040884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.040852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-config\") pod \"eef65afd-4517-4baa-936e-fc8dfb8eadef\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") "
Jul 28 17:12:32.041047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.040919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sblq\" (UniqueName: \"kubernetes.io/projected/eef65afd-4517-4baa-936e-fc8dfb8eadef-kube-api-access-5sblq\") pod \"eef65afd-4517-4baa-936e-fc8dfb8eadef\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") "
Jul 28 17:12:32.041047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.040937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-trusted-ca-bundle\") pod \"eef65afd-4517-4baa-936e-fc8dfb8eadef\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") "
Jul 28 17:12:32.041047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.040968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-oauth-config\") pod \"eef65afd-4517-4baa-936e-fc8dfb8eadef\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") "
Jul 28 17:12:32.041047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.040994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-serving-cert\") pod \"eef65afd-4517-4baa-936e-fc8dfb8eadef\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") "
Jul 28 17:12:32.041047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.041020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-oauth-serving-cert\") pod \"eef65afd-4517-4baa-936e-fc8dfb8eadef\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") "
Jul 28 17:12:32.041047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.041038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-service-ca\") pod \"eef65afd-4517-4baa-936e-fc8dfb8eadef\" (UID: \"eef65afd-4517-4baa-936e-fc8dfb8eadef\") "
Jul 28 17:12:32.041475 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.041443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-config" (OuterVolumeSpecName: "console-config") pod "eef65afd-4517-4baa-936e-fc8dfb8eadef" (UID: "eef65afd-4517-4baa-936e-fc8dfb8eadef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:32.041616 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.041582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-service-ca" (OuterVolumeSpecName: "service-ca") pod "eef65afd-4517-4baa-936e-fc8dfb8eadef" (UID: "eef65afd-4517-4baa-936e-fc8dfb8eadef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:32.041679 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.041608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eef65afd-4517-4baa-936e-fc8dfb8eadef" (UID: "eef65afd-4517-4baa-936e-fc8dfb8eadef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:32.041679 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.041586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eef65afd-4517-4baa-936e-fc8dfb8eadef" (UID: "eef65afd-4517-4baa-936e-fc8dfb8eadef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:32.043230 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.043201    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eef65afd-4517-4baa-936e-fc8dfb8eadef" (UID: "eef65afd-4517-4baa-936e-fc8dfb8eadef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:32.043365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.043283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eef65afd-4517-4baa-936e-fc8dfb8eadef" (UID: "eef65afd-4517-4baa-936e-fc8dfb8eadef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:32.043365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.043292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef65afd-4517-4baa-936e-fc8dfb8eadef-kube-api-access-5sblq" (OuterVolumeSpecName: "kube-api-access-5sblq") pod "eef65afd-4517-4baa-936e-fc8dfb8eadef" (UID: "eef65afd-4517-4baa-936e-fc8dfb8eadef"). InnerVolumeSpecName "kube-api-access-5sblq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:32.141917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.141836    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sblq\" (UniqueName: \"kubernetes.io/projected/eef65afd-4517-4baa-936e-fc8dfb8eadef-kube-api-access-5sblq\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:12:32.141917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.141865    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-trusted-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:12:32.141917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.141877    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-oauth-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:12:32.141917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.141889    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-serving-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:12:32.141917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.141901    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-oauth-serving-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:12:32.141917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.141913    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:12:32.142195 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.141924    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eef65afd-4517-4baa-936e-fc8dfb8eadef-console-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:12:32.899646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.899611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7898c74669-lj75b" event={"ID":"eef65afd-4517-4baa-936e-fc8dfb8eadef","Type":"ContainerDied","Data":"d19d4c1561fed55353968ea9f31fc608287059c22438a325b662330670474bbb"}
Jul 28 17:12:32.899646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.899632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7898c74669-lj75b"
Jul 28 17:12:32.900126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.899682    2578 scope.go:117] "RemoveContainer" containerID="542e0fdea953d8ab962455697a0b9c7af59619e52335450d92054a38fdf8a1a3"
Jul 28 17:12:32.921923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.921900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7898c74669-lj75b"]
Jul 28 17:12:32.926128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:32.926108    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7898c74669-lj75b"]
Jul 28 17:12:33.362391 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:33.362357    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef65afd-4517-4baa-936e-fc8dfb8eadef" path="/var/lib/kubelet/pods/eef65afd-4517-4baa-936e-fc8dfb8eadef/volumes"
Jul 28 17:12:34.608467 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:34.608438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:12:34.612205 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:34.612185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f48c497b6-7hqb5"
Jul 28 17:12:56.652861 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:56.652814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:56.670752 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:56.670721    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:57.004697 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:12:57.004628    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:09.257030 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:09.256984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:09.454988 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:09.454956    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:10.303264 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:10.303227    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:10.304175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:10.304138    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="alertmanager" containerID="cri-o://5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5" gracePeriod=120
Jul 28 17:13:10.304745 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:10.304361    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-metric" containerID="cri-o://0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e" gracePeriod=120
Jul 28 17:13:10.306060 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:10.304537    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="prom-label-proxy" containerID="cri-o://00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0" gracePeriod=120
Jul 28 17:13:10.306859 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:10.304559    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-web" containerID="cri-o://bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902" gracePeriod=120
Jul 28 17:13:10.307028 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:10.304575    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy" containerID="cri-o://729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309" gracePeriod=120
Jul 28 17:13:10.307179 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:10.304592    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="config-reloader" containerID="cri-o://b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28" gracePeriod=120
Jul 28 17:13:11.036733 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036698    2578 generic.go:356] "Generic (PLEG): container finished" podID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerID="00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0" exitCode=0
Jul 28 17:13:11.036733 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036724    2578 generic.go:356] "Generic (PLEG): container finished" podID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerID="0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e" exitCode=0
Jul 28 17:13:11.036733 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036731    2578 generic.go:356] "Generic (PLEG): container finished" podID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerID="729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309" exitCode=0
Jul 28 17:13:11.036733 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036737    2578 generic.go:356] "Generic (PLEG): container finished" podID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerID="b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28" exitCode=0
Jul 28 17:13:11.036733 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036742    2578 generic.go:356] "Generic (PLEG): container finished" podID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerID="5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5" exitCode=0
Jul 28 17:13:11.037011 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0"}
Jul 28 17:13:11.037011 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e"}
Jul 28 17:13:11.037011 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309"}
Jul 28 17:13:11.037011 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28"}
Jul 28 17:13:11.037011 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.036914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5"}
Jul 28 17:13:11.559485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.559463    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:11.644616 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-out\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-web-config\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-metrics-client-ca\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-tls-assets\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-volume\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj9ct\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-kube-api-access-tj9ct\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644997 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-cluster-tls-config\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644997 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.644997 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.644854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.645160 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.645090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:11.646162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.645437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.646162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.645507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.646162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.645550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-main-db\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.646162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.645586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-main-tls\") pod \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\" (UID: \"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6\") "
Jul 28 17:13:11.646162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.645831    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-metrics-client-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.647272 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.647243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:13:11.647512 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.647489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:11.647879 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.647829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:11.648221 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.648180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:11.648609 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.648566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:11.648731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.648690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-out" (OuterVolumeSpecName: "config-out") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:13:11.648920 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.648833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-volume" (OuterVolumeSpecName: "config-volume") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:11.648975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.648935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-kube-api-access-tj9ct" (OuterVolumeSpecName: "kube-api-access-tj9ct") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "kube-api-access-tj9ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:11.649082 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.649057    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:11.649372 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.649352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:11.651818 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.651800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:11.658730 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.658666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-web-config" (OuterVolumeSpecName: "web-config") pod "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" (UID: "53408a8c-ffb1-41ce-82fb-7ed0cd3312f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:11.746687 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746651    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-cluster-tls-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746687 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746681    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746687 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746692    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746702    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746712    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746721    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-alertmanager-main-db\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746730    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-secret-alertmanager-main-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746739    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-out\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746748    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-web-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746755    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-tls-assets\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746764    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-config-volume\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:11.746905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:11.746772    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj9ct\" (UniqueName: \"kubernetes.io/projected/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6-kube-api-access-tj9ct\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:12.045236 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.045162    2578 generic.go:356] "Generic (PLEG): container finished" podID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerID="bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902" exitCode=0
Jul 28 17:13:12.045236 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.045219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902"}
Jul 28 17:13:12.045423 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.045238    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.045423 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.045275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"53408a8c-ffb1-41ce-82fb-7ed0cd3312f6","Type":"ContainerDied","Data":"bd896aec85a78d75e0f61d591e4dbe76fbacca765759ee059031c307ef637cbd"}
Jul 28 17:13:12.045423 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.045326    2578 scope.go:117] "RemoveContainer" containerID="00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0"
Jul 28 17:13:12.052218 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.052196    2578 scope.go:117] "RemoveContainer" containerID="0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e"
Jul 28 17:13:12.058686 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.058670    2578 scope.go:117] "RemoveContainer" containerID="729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309"
Jul 28 17:13:12.064862 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.064847    2578 scope.go:117] "RemoveContainer" containerID="bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902"
Jul 28 17:13:12.069803 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.069781    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:12.072105 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.072081    2578 scope.go:117] "RemoveContainer" containerID="b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28"
Jul 28 17:13:12.075223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.075199    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:12.079924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.079909    2578 scope.go:117] "RemoveContainer" containerID="5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5"
Jul 28 17:13:12.086206 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.086189    2578 scope.go:117] "RemoveContainer" containerID="28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2"
Jul 28 17:13:12.092359 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.092345    2578 scope.go:117] "RemoveContainer" containerID="00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0"
Jul 28 17:13:12.092621 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:12.092603    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0\": container with ID starting with 00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0 not found: ID does not exist" containerID="00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0"
Jul 28 17:13:12.092673 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.092629    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0"} err="failed to get container status \"00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0\": rpc error: code = NotFound desc = could not find container \"00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0\": container with ID starting with 00d14eebc34c76ffd72373a504018f924d87015dd6dcf5ce80695abb5dbdd4f0 not found: ID does not exist"
Jul 28 17:13:12.092673 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.092660    2578 scope.go:117] "RemoveContainer" containerID="0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e"
Jul 28 17:13:12.092867 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:12.092850    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e\": container with ID starting with 0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e not found: ID does not exist" containerID="0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e"
Jul 28 17:13:12.092906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.092873    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e"} err="failed to get container status \"0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e\": rpc error: code = NotFound desc = could not find container \"0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e\": container with ID starting with 0dda67218aec96597565ec97abef54e1ed4a7691908cee4225a2ac56c241ea2e not found: ID does not exist"
Jul 28 17:13:12.092906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.092888    2578 scope.go:117] "RemoveContainer" containerID="729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309"
Jul 28 17:13:12.093096 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:12.093079    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309\": container with ID starting with 729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309 not found: ID does not exist" containerID="729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309"
Jul 28 17:13:12.093137 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093102    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309"} err="failed to get container status \"729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309\": rpc error: code = NotFound desc = could not find container \"729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309\": container with ID starting with 729f1d203caa39f850ff62ed68565895a4069a4e9425a6d75a81cf2f7c30a309 not found: ID does not exist"
Jul 28 17:13:12.093137 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093118    2578 scope.go:117] "RemoveContainer" containerID="bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902"
Jul 28 17:13:12.093366 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:12.093346    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902\": container with ID starting with bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902 not found: ID does not exist" containerID="bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902"
Jul 28 17:13:12.093411 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093370    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902"} err="failed to get container status \"bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902\": rpc error: code = NotFound desc = could not find container \"bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902\": container with ID starting with bd5b321f99caf97c34edcde3f4ae64429f4ee8aee6db41189519823ec5051902 not found: ID does not exist"
Jul 28 17:13:12.093411 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093384    2578 scope.go:117] "RemoveContainer" containerID="b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28"
Jul 28 17:13:12.093573 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:12.093560    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28\": container with ID starting with b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28 not found: ID does not exist" containerID="b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28"
Jul 28 17:13:12.093614 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093575    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28"} err="failed to get container status \"b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28\": rpc error: code = NotFound desc = could not find container \"b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28\": container with ID starting with b788c5725a731a0ee32eb926c466bf8cf73f8a965db65e586e0a0cfe63500a28 not found: ID does not exist"
Jul 28 17:13:12.093614 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093586    2578 scope.go:117] "RemoveContainer" containerID="5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5"
Jul 28 17:13:12.093784 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:12.093770    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5\": container with ID starting with 5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5 not found: ID does not exist" containerID="5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5"
Jul 28 17:13:12.093823 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093787    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5"} err="failed to get container status \"5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5\": rpc error: code = NotFound desc = could not find container \"5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5\": container with ID starting with 5ab18a75b9bc792c6411759b988ae06e38c4cb04a302cad89483a2c39430d7e5 not found: ID does not exist"
Jul 28 17:13:12.093823 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093799    2578 scope.go:117] "RemoveContainer" containerID="28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2"
Jul 28 17:13:12.093972 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:12.093958    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2\": container with ID starting with 28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2 not found: ID does not exist" containerID="28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2"
Jul 28 17:13:12.094015 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.093976    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2"} err="failed to get container status \"28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2\": rpc error: code = NotFound desc = could not find container \"28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2\": container with ID starting with 28a729dfd47e4d19f3d1738eededdac131be1824dc5ff5190ff11b015f1558d2 not found: ID does not exist"
Jul 28 17:13:12.108643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108622    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:12.108851 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108840    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-metric"
Jul 28 17:13:12.108893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108853    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-metric"
Jul 28 17:13:12.108893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108868    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-web"
Jul 28 17:13:12.108893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108874    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-web"
Jul 28 17:13:12.108893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108882    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="prom-label-proxy"
Jul 28 17:13:12.108893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108888    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="prom-label-proxy"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108896    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef65afd-4517-4baa-936e-fc8dfb8eadef" containerName="console"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108901    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef65afd-4517-4baa-936e-fc8dfb8eadef" containerName="console"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108906    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108918    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="init-config-reloader"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="init-config-reloader"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108931    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="alertmanager"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108936    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="alertmanager"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108943    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="config-reloader"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108948    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="config-reloader"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108983    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef65afd-4517-4baa-936e-fc8dfb8eadef" containerName="console"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108990    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="alertmanager"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.108995    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-web"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.109000    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.109005    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="kube-rbac-proxy-metric"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.109011    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="prom-label-proxy"
Jul 28 17:13:12.109046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.109016    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" containerName="config-reloader"
Jul 28 17:13:12.114161 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.112730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.116172 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:13:12.116415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:13:12.116542 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116421    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mfpvw\""
Jul 28 17:13:12.116542 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:13:12.116790 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:13:12.116929 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:13:12.116929 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116844    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:13:12.116929 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.116853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:13:12.117612 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.117596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:13:12.122376 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.122358    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:13:12.127433 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.127416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:12.250715 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-web-config\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.250715 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/48ba9010-ee07-4cae-8dbd-a770f3437eec-kube-api-access-tbslp\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.250924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48ba9010-ee07-4cae-8dbd-a770f3437eec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.250924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.250924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.250924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.250924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48ba9010-ee07-4cae-8dbd-a770f3437eec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.251077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ba9010-ee07-4cae-8dbd-a770f3437eec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.251077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48ba9010-ee07-4cae-8dbd-a770f3437eec-config-out\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.251077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.250996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.251077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.251011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-config-volume\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.251202 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.251085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ba9010-ee07-4cae-8dbd-a770f3437eec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.251202 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.251124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352070 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/48ba9010-ee07-4cae-8dbd-a770f3437eec-kube-api-access-tbslp\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352176 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48ba9010-ee07-4cae-8dbd-a770f3437eec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352176 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352176 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352292 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352292 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48ba9010-ee07-4cae-8dbd-a770f3437eec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ba9010-ee07-4cae-8dbd-a770f3437eec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48ba9010-ee07-4cae-8dbd-a770f3437eec-config-out\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-config-volume\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ba9010-ee07-4cae-8dbd-a770f3437eec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-web-config\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.353045 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.352728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/48ba9010-ee07-4cae-8dbd-a770f3437eec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.353143 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.353119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ba9010-ee07-4cae-8dbd-a770f3437eec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355181 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48ba9010-ee07-4cae-8dbd-a770f3437eec-config-out\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355277 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48ba9010-ee07-4cae-8dbd-a770f3437eec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355277 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355419 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355487 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-config-volume\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355668 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-web-config\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355782 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.355928 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.355908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ba9010-ee07-4cae-8dbd-a770f3437eec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.356130 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.356117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.357021 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.356996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/48ba9010-ee07-4cae-8dbd-a770f3437eec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.361415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.361394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbslp\" (UniqueName: \"kubernetes.io/projected/48ba9010-ee07-4cae-8dbd-a770f3437eec-kube-api-access-tbslp\") pod \"alertmanager-main-0\" (UID: \"48ba9010-ee07-4cae-8dbd-a770f3437eec\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.423381 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.423363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:12.552596 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:12.552563    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:12.555323 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:13:12.555281    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ba9010_ee07_4cae_8dbd_a770f3437eec.slice/crio-75c6daab3bac95c7fff8f5064cbc8e5d8dea0cba987bbe89892729f66147904a WatchSource:0}: Error finding container 75c6daab3bac95c7fff8f5064cbc8e5d8dea0cba987bbe89892729f66147904a: Status 404 returned error can't find the container with id 75c6daab3bac95c7fff8f5064cbc8e5d8dea0cba987bbe89892729f66147904a
Jul 28 17:13:13.050424 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:13.050335    2578 generic.go:356] "Generic (PLEG): container finished" podID="48ba9010-ee07-4cae-8dbd-a770f3437eec" containerID="8156492bccd28ab55e46ab9524457937eabe77171181e48b721740fa533db4cf" exitCode=0
Jul 28 17:13:13.050424 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:13.050376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerDied","Data":"8156492bccd28ab55e46ab9524457937eabe77171181e48b721740fa533db4cf"}
Jul 28 17:13:13.050424 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:13.050426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerStarted","Data":"75c6daab3bac95c7fff8f5064cbc8e5d8dea0cba987bbe89892729f66147904a"}
Jul 28 17:13:13.362978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:13.362953    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53408a8c-ffb1-41ce-82fb-7ed0cd3312f6" path="/var/lib/kubelet/pods/53408a8c-ffb1-41ce-82fb-7ed0cd3312f6/volumes"
Jul 28 17:13:14.061863 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.061824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerStarted","Data":"486debfb8b819ee92c854bc6108b545c8516a6d468e39969273268ac223b10e0"}
Jul 28 17:13:14.062220 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.061934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerStarted","Data":"b7e64b9fd5e6280c5a55b2bfe465327e29e7538a27cbf18f754752bcf1917724"}
Jul 28 17:13:14.062220 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.061975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerStarted","Data":"8f6812c582c30def85f3c62c8d08f7e18e6bd92bbaae98078ae0a6fe73e9235d"}
Jul 28 17:13:14.062220 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.062012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerStarted","Data":"2d9938b7d544fa945881a0a3c96bc3ca2aafa549ebfc94b67fe37b0ec7779081"}
Jul 28 17:13:14.062220 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.062047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerStarted","Data":"34baf82995cab11bfd6a66df6650a2a8f4c9c0d1ac5730b4bc6c3a47cf8563c1"}
Jul 28 17:13:14.062220 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.062089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"48ba9010-ee07-4cae-8dbd-a770f3437eec","Type":"ContainerStarted","Data":"4086f4aa52c06fb3143b8087f21ae484ef6eed6e6b807c84a0bc7d51e18397fe"}
Jul 28 17:13:14.114221 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.114168    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.098663529 podStartE2EDuration="2.114153236s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.489707ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:14.11170976 +0000 UTC m=+149.277345607" watchObservedRunningTime="2026-07-28 17:13:14.114153236 +0000 UTC m=+149.279789070"
Jul 28 17:13:14.602424 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.602392    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:13:14.602714 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.602683    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="prometheus" containerID="cri-o://9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03" gracePeriod=600
Jul 28 17:13:14.602874 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.602713    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy" containerID="cri-o://2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63" gracePeriod=600
Jul 28 17:13:14.602874 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.602753    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="config-reloader" containerID="cri-o://2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0" gracePeriod=600
Jul 28 17:13:14.602874 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.602759    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-web" containerID="cri-o://0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540" gracePeriod=600
Jul 28 17:13:14.602874 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.602756    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="thanos-sidecar" containerID="cri-o://838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018" gracePeriod=600
Jul 28 17:13:14.603092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.602870    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083" gracePeriod=600
Jul 28 17:13:14.853971 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.853908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:14.976430 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-trusted-ca-bundle\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976430 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z4wz\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-kube-api-access-2z4wz\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-kubelet-serving-ca-bundle\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-metrics-client-certs\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-config\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976533    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-serving-certs-ca-bundle\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976563    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-1\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976580    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-web-config\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-db\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.976672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-kube-rbac-proxy\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-metrics-client-ca\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976708    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-0\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-tls\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976849    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-grpc-tls\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-config-out\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976913    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.976916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-thanos-prometheus-http-client-file\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-tls-assets\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-2\") pod \"7ebf3fea-6e9c-4459-b332-03ad67904f88\" (UID: \"7ebf3fea-6e9c-4459-b332-03ad67904f88\") "
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977385    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-metrics-client-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977405    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977423    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977437    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977451    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:14.977651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.977479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:14.978496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.978466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:13:14.979100 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.979067    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-config" (OuterVolumeSpecName: "config") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.979208 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.979187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.979352 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.979329    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.979678 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.979642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-kube-api-access-2z4wz" (OuterVolumeSpecName: "kube-api-access-2z4wz") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "kube-api-access-2z4wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:14.980040 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.980012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:14.980550 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.980512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:14.981223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.981180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.981326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.981247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.981523 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.981495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-config-out" (OuterVolumeSpecName: "config-out") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:13:14.981640 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.981571    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.981787 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.981760    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.982003 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.981982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:14.988709 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:14.988690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-web-config" (OuterVolumeSpecName: "web-config") pod "7ebf3fea-6e9c-4459-b332-03ad67904f88" (UID: "7ebf3fea-6e9c-4459-b332-03ad67904f88"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:15.068777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068751    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083" exitCode=0
Jul 28 17:13:15.068777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068772    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63" exitCode=0
Jul 28 17:13:15.068777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068780    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540" exitCode=0
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068786    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018" exitCode=0
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068791    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0" exitCode=0
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068797    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03" exitCode=0
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068831    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"}
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"}
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"}
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.068981    2578 scope.go:117] "RemoveContainer" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.069011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"}
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.069051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"}
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.069093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"}
Jul 28 17:13:15.069184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.069134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ebf3fea-6e9c-4459-b332-03ad67904f88","Type":"ContainerDied","Data":"d230ef9d3b38f9ddf3dde0e650b208f55185b21b36f3ad37ae2d12a46152a6d7"}
Jul 28 17:13:15.076521 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.076495    2578 scope.go:117] "RemoveContainer" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.077775 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077752    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-kube-rbac-proxy\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.077854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077780    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.077854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077795    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.077854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077810    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.077854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077824    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.077854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077837    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-grpc-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.077854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077851    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-config-out\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077863    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077877    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-tls-assets\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077890    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077916    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z4wz\" (UniqueName: \"kubernetes.io/projected/7ebf3fea-6e9c-4459-b332-03ad67904f88-kube-api-access-2z4wz\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077929    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-secret-metrics-client-certs\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077941    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077953    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ebf3fea-6e9c-4459-b332-03ad67904f88-web-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.078027 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.077979    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ebf3fea-6e9c-4459-b332-03ad67904f88-prometheus-k8s-db\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:13:15.082906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.082883    2578 scope.go:117] "RemoveContainer" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.088975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.088956    2578 scope.go:117] "RemoveContainer" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.094333 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.094311    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:13:15.095554 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.095373    2578 scope.go:117] "RemoveContainer" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.102099 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.102078    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:13:15.102809 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.102795    2578 scope.go:117] "RemoveContainer" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.109341 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.109325    2578 scope.go:117] "RemoveContainer" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.115236 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.115221    2578 scope.go:117] "RemoveContainer" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.115497 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:15.115480    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": container with ID starting with ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083 not found: ID does not exist" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.115563 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.115509    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"} err="failed to get container status \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": rpc error: code = NotFound desc = could not find container \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": container with ID starting with ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083 not found: ID does not exist"
Jul 28 17:13:15.115563 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.115533    2578 scope.go:117] "RemoveContainer" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.115777 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:15.115760    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": container with ID starting with 2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63 not found: ID does not exist" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.115812 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.115782    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"} err="failed to get container status \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": rpc error: code = NotFound desc = could not find container \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": container with ID starting with 2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63 not found: ID does not exist"
Jul 28 17:13:15.115812 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.115800    2578 scope.go:117] "RemoveContainer" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.116016 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:15.115999    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": container with ID starting with 0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540 not found: ID does not exist" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.116064 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116021    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"} err="failed to get container status \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": rpc error: code = NotFound desc = could not find container \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": container with ID starting with 0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540 not found: ID does not exist"
Jul 28 17:13:15.116064 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116035    2578 scope.go:117] "RemoveContainer" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.116253 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:15.116236    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": container with ID starting with 838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018 not found: ID does not exist" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.116333 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"} err="failed to get container status \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": rpc error: code = NotFound desc = could not find container \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": container with ID starting with 838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018 not found: ID does not exist"
Jul 28 17:13:15.116333 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116280    2578 scope.go:117] "RemoveContainer" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.116556 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:15.116539    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": container with ID starting with 2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0 not found: ID does not exist" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.116601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116559    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"} err="failed to get container status \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": rpc error: code = NotFound desc = could not find container \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": container with ID starting with 2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0 not found: ID does not exist"
Jul 28 17:13:15.116601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116575    2578 scope.go:117] "RemoveContainer" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.116780 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:15.116764    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": container with ID starting with 9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03 not found: ID does not exist" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.116841 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116786    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"} err="failed to get container status \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": rpc error: code = NotFound desc = could not find container \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": container with ID starting with 9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03 not found: ID does not exist"
Jul 28 17:13:15.116841 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.116805    2578 scope.go:117] "RemoveContainer" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.117026 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:13:15.117010    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": container with ID starting with 37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314 not found: ID does not exist" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.117071 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117031    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"} err="failed to get container status \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": rpc error: code = NotFound desc = could not find container \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": container with ID starting with 37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314 not found: ID does not exist"
Jul 28 17:13:15.117071 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117046    2578 scope.go:117] "RemoveContainer" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.117234 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117215    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"} err="failed to get container status \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": rpc error: code = NotFound desc = could not find container \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": container with ID starting with ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083 not found: ID does not exist"
Jul 28 17:13:15.117276 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117235    2578 scope.go:117] "RemoveContainer" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.117537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117509    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"} err="failed to get container status \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": rpc error: code = NotFound desc = could not find container \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": container with ID starting with 2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63 not found: ID does not exist"
Jul 28 17:13:15.117638 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117538    2578 scope.go:117] "RemoveContainer" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.117777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117759    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"} err="failed to get container status \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": rpc error: code = NotFound desc = could not find container \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": container with ID starting with 0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540 not found: ID does not exist"
Jul 28 17:13:15.117843 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.117778    2578 scope.go:117] "RemoveContainer" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.118081 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118064    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"} err="failed to get container status \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": rpc error: code = NotFound desc = could not find container \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": container with ID starting with 838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018 not found: ID does not exist"
Jul 28 17:13:15.118152 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118082    2578 scope.go:117] "RemoveContainer" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.118336 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118317    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"} err="failed to get container status \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": rpc error: code = NotFound desc = could not find container \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": container with ID starting with 2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0 not found: ID does not exist"
Jul 28 17:13:15.118395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118337    2578 scope.go:117] "RemoveContainer" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.118559 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118541    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"} err="failed to get container status \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": rpc error: code = NotFound desc = could not find container \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": container with ID starting with 9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03 not found: ID does not exist"
Jul 28 17:13:15.118618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118560    2578 scope.go:117] "RemoveContainer" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.118752 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118735    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"} err="failed to get container status \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": rpc error: code = NotFound desc = could not find container \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": container with ID starting with 37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314 not found: ID does not exist"
Jul 28 17:13:15.118817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118753    2578 scope.go:117] "RemoveContainer" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.118942 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118925    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"} err="failed to get container status \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": rpc error: code = NotFound desc = could not find container \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": container with ID starting with ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083 not found: ID does not exist"
Jul 28 17:13:15.118998 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.118943    2578 scope.go:117] "RemoveContainer" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.119201 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119182    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"} err="failed to get container status \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": rpc error: code = NotFound desc = could not find container \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": container with ID starting with 2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63 not found: ID does not exist"
Jul 28 17:13:15.119250 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119202    2578 scope.go:117] "RemoveContainer" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.119437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119419    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"} err="failed to get container status \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": rpc error: code = NotFound desc = could not find container \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": container with ID starting with 0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540 not found: ID does not exist"
Jul 28 17:13:15.119481 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119438    2578 scope.go:117] "RemoveContainer" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.119644 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119627    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"} err="failed to get container status \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": rpc error: code = NotFound desc = could not find container \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": container with ID starting with 838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018 not found: ID does not exist"
Jul 28 17:13:15.119683 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119644    2578 scope.go:117] "RemoveContainer" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.119839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119822    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"} err="failed to get container status \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": rpc error: code = NotFound desc = could not find container \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": container with ID starting with 2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0 not found: ID does not exist"
Jul 28 17:13:15.119839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119836    2578 scope.go:117] "RemoveContainer" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.120000 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.119981    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"} err="failed to get container status \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": rpc error: code = NotFound desc = could not find container \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": container with ID starting with 9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03 not found: ID does not exist"
Jul 28 17:13:15.120065 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120002    2578 scope.go:117] "RemoveContainer" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.120209 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120191    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"} err="failed to get container status \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": rpc error: code = NotFound desc = could not find container \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": container with ID starting with 37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314 not found: ID does not exist"
Jul 28 17:13:15.120251 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120210    2578 scope.go:117] "RemoveContainer" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.120436 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120422    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"} err="failed to get container status \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": rpc error: code = NotFound desc = could not find container \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": container with ID starting with ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083 not found: ID does not exist"
Jul 28 17:13:15.120485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120436    2578 scope.go:117] "RemoveContainer" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.120639 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120624    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"} err="failed to get container status \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": rpc error: code = NotFound desc = could not find container \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": container with ID starting with 2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63 not found: ID does not exist"
Jul 28 17:13:15.120639 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120637    2578 scope.go:117] "RemoveContainer" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.120825 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120806    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"} err="failed to get container status \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": rpc error: code = NotFound desc = could not find container \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": container with ID starting with 0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540 not found: ID does not exist"
Jul 28 17:13:15.120869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.120827    2578 scope.go:117] "RemoveContainer" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.121026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121008    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"} err="failed to get container status \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": rpc error: code = NotFound desc = could not find container \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": container with ID starting with 838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018 not found: ID does not exist"
Jul 28 17:13:15.121095 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121027    2578 scope.go:117] "RemoveContainer" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.121236 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121216    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"} err="failed to get container status \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": rpc error: code = NotFound desc = could not find container \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": container with ID starting with 2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0 not found: ID does not exist"
Jul 28 17:13:15.121280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121237    2578 scope.go:117] "RemoveContainer" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.121435 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121417    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"} err="failed to get container status \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": rpc error: code = NotFound desc = could not find container \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": container with ID starting with 9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03 not found: ID does not exist"
Jul 28 17:13:15.121483 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121436    2578 scope.go:117] "RemoveContainer" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.121630 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121613    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"} err="failed to get container status \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": rpc error: code = NotFound desc = could not find container \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": container with ID starting with 37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314 not found: ID does not exist"
Jul 28 17:13:15.121688 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121632    2578 scope.go:117] "RemoveContainer" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.121853 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121838    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"} err="failed to get container status \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": rpc error: code = NotFound desc = could not find container \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": container with ID starting with ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083 not found: ID does not exist"
Jul 28 17:13:15.121897 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.121853    2578 scope.go:117] "RemoveContainer" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.122044 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122027    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"} err="failed to get container status \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": rpc error: code = NotFound desc = could not find container \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": container with ID starting with 2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63 not found: ID does not exist"
Jul 28 17:13:15.122082 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122045    2578 scope.go:117] "RemoveContainer" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.122213 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122197    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"} err="failed to get container status \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": rpc error: code = NotFound desc = could not find container \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": container with ID starting with 0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540 not found: ID does not exist"
Jul 28 17:13:15.122254 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122214    2578 scope.go:117] "RemoveContainer" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.122415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122401    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"} err="failed to get container status \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": rpc error: code = NotFound desc = could not find container \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": container with ID starting with 838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018 not found: ID does not exist"
Jul 28 17:13:15.122415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122414    2578 scope.go:117] "RemoveContainer" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.122598 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122581    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"} err="failed to get container status \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": rpc error: code = NotFound desc = could not find container \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": container with ID starting with 2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0 not found: ID does not exist"
Jul 28 17:13:15.122653 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122599    2578 scope.go:117] "RemoveContainer" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.122762 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122748    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"} err="failed to get container status \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": rpc error: code = NotFound desc = could not find container \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": container with ID starting with 9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03 not found: ID does not exist"
Jul 28 17:13:15.122802 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122762    2578 scope.go:117] "RemoveContainer" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.122960 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122944    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"} err="failed to get container status \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": rpc error: code = NotFound desc = could not find container \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": container with ID starting with 37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314 not found: ID does not exist"
Jul 28 17:13:15.122995 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.122963    2578 scope.go:117] "RemoveContainer" containerID="ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"
Jul 28 17:13:15.123160 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123143    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083"} err="failed to get container status \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": rpc error: code = NotFound desc = could not find container \"ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083\": container with ID starting with ca74101e591e5a0ef063f0d754a7a72d3dbfaa11712771aa46cabcaf5fb5b083 not found: ID does not exist"
Jul 28 17:13:15.123200 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123160    2578 scope.go:117] "RemoveContainer" containerID="2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"
Jul 28 17:13:15.123365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123350    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63"} err="failed to get container status \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": rpc error: code = NotFound desc = could not find container \"2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63\": container with ID starting with 2a520cb8801fae66ffdbab42997c720ac6642729746cbe2b69bab2b55fd16a63 not found: ID does not exist"
Jul 28 17:13:15.123365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123365    2578 scope.go:117] "RemoveContainer" containerID="0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"
Jul 28 17:13:15.123543 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123527    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540"} err="failed to get container status \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": rpc error: code = NotFound desc = could not find container \"0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540\": container with ID starting with 0135584b89b42d4da948362aaeaadbee689bad339b82d3575f262e76a77e5540 not found: ID does not exist"
Jul 28 17:13:15.123588 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123544    2578 scope.go:117] "RemoveContainer" containerID="838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"
Jul 28 17:13:15.123727 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123702    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018"} err="failed to get container status \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": rpc error: code = NotFound desc = could not find container \"838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018\": container with ID starting with 838f9d4e2082575e8395a5e3c8ce8512e9f0eaf8af32b08c8ce9647fc79bc018 not found: ID does not exist"
Jul 28 17:13:15.123727 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123721    2578 scope.go:117] "RemoveContainer" containerID="2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"
Jul 28 17:13:15.123942 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123924    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0"} err="failed to get container status \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": rpc error: code = NotFound desc = could not find container \"2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0\": container with ID starting with 2ef392b495fd16287084c9dbc4af6c1ae6918cfe76af39a85f7ff977c4c524f0 not found: ID does not exist"
Jul 28 17:13:15.124017 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.123943    2578 scope.go:117] "RemoveContainer" containerID="9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"
Jul 28 17:13:15.124165 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.124151    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03"} err="failed to get container status \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": rpc error: code = NotFound desc = could not find container \"9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03\": container with ID starting with 9960973d5b847e7887a53f574c23f875301fe9bdbe0eb7659281c9c4efeebb03 not found: ID does not exist"
Jul 28 17:13:15.124210 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.124166    2578 scope.go:117] "RemoveContainer" containerID="37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"
Jul 28 17:13:15.124369 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.124352    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314"} err="failed to get container status \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": rpc error: code = NotFound desc = could not find container \"37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314\": container with ID starting with 37a759812a3f40676e87595fbf6c220708a1fa4a78b4a6fe6c1a94a5bbde2314 not found: ID does not exist"
Jul 28 17:13:15.127157 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127138    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:13:15.127406 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127393    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-web"
Jul 28 17:13:15.127450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127408    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-web"
Jul 28 17:13:15.127450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127416    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="thanos-sidecar"
Jul 28 17:13:15.127450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127422    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="thanos-sidecar"
Jul 28 17:13:15.127450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127432    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="init-config-reloader"
Jul 28 17:13:15.127450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127439    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="init-config-reloader"
Jul 28 17:13:15.127450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127445    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy"
Jul 28 17:13:15.127450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127451    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127461    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="config-reloader"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127469    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="config-reloader"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127481    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-thanos"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127486    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-thanos"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127494    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="prometheus"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127498    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="prometheus"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-web"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127548    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127555    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="config-reloader"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127560    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="kube-rbac-proxy-thanos"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127566    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="prometheus"
Jul 28 17:13:15.127643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.127572    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" containerName="thanos-sidecar"
Jul 28 17:13:15.131822 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.130963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.134316 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.134280    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:13:15.134381 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.134293    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:13:15.134381 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.134352    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:13:15.135428 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135394    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9vkf9f9gt4dg5\""
Jul 28 17:13:15.135707 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:13:15.135788 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135730    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:13:15.135788 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135694    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:13:15.135788 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135753    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:13:15.135992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sg5kw\""
Jul 28 17:13:15.135992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135891    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:13:15.135992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.135972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:13:15.136147 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.136074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:13:15.136196 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.136183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:13:15.136408 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.136392    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:13:15.139235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.139214    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:13:15.141237 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.141220    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:13:15.154381 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.154359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:13:15.280911 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.280881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.280911 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.280918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.280946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.280989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-config\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-web-config\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28dp\" (UniqueName: \"kubernetes.io/projected/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-kube-api-access-m28dp\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281690 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281690 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281690 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281690 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281690 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-config-out\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.281690 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.281615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.364291 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.364226    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ebf3fea-6e9c-4459-b332-03ad67904f88" path="/var/lib/kubelet/pods/7ebf3fea-6e9c-4459-b332-03ad67904f88/volumes"
Jul 28 17:13:15.382125 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382232 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382232 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382232 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382232 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-config-out\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-config\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-web-config\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.382860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28dp\" (UniqueName: \"kubernetes.io/projected/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-kube-api-access-m28dp\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.383250 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.382917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.383250 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.383014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.383613 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.383587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.383826 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.383801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.385628 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.385268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.385628 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.385337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-config\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.385628 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.385378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.385628 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.385593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-config-out\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.385902 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.385735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.385962 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.385927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.386073 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.386050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.386207 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.386185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.386330 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.386266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.387536 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.387508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.387741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.387723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.387814 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.387797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-web-config\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.387905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.387880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.388230 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.388216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.391493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.391477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28dp\" (UniqueName: \"kubernetes.io/projected/bdfd7f79-2cb6-423d-ad0a-86aa70b53102-kube-api-access-m28dp\") pod \"prometheus-k8s-0\" (UID: \"bdfd7f79-2cb6-423d-ad0a-86aa70b53102\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.440970 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.440947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:15.573023 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:15.572847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:13:15.574776 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:13:15.574754    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfd7f79_2cb6_423d_ad0a_86aa70b53102.slice/crio-7205fa85f05daad3b8a2e445d5da03a9cb704ac9e0c3bd5b7b95a5211808972a WatchSource:0}: Error finding container 7205fa85f05daad3b8a2e445d5da03a9cb704ac9e0c3bd5b7b95a5211808972a: Status 404 returned error can't find the container with id 7205fa85f05daad3b8a2e445d5da03a9cb704ac9e0c3bd5b7b95a5211808972a
Jul 28 17:13:16.074658 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:16.074628    2578 generic.go:356] "Generic (PLEG): container finished" podID="bdfd7f79-2cb6-423d-ad0a-86aa70b53102" containerID="2303d96fd15f3a4d131a4d7e20c1f3639584d232f04106002d381c328c77ed39" exitCode=0
Jul 28 17:13:16.075004 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:16.074697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerDied","Data":"2303d96fd15f3a4d131a4d7e20c1f3639584d232f04106002d381c328c77ed39"}
Jul 28 17:13:16.075004 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:16.074752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerStarted","Data":"7205fa85f05daad3b8a2e445d5da03a9cb704ac9e0c3bd5b7b95a5211808972a"}
Jul 28 17:13:17.085035 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:17.084996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerStarted","Data":"f6f62b5c50781cc953344d5daedb1f8c8f0047f4a8708040f04f423d79486f98"}
Jul 28 17:13:17.085391 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:17.085107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerStarted","Data":"76a4680f2e92a81369bce4afbe49ca7c239d06b4e3ee24e2f47e829be2caea53"}
Jul 28 17:13:17.085391 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:17.085147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerStarted","Data":"2c91ed5b8e38fd7481c777d80a010b79f42b446d628784a97c5ed5323a07c4e0"}
Jul 28 17:13:17.085391 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:17.085185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerStarted","Data":"40c2c790fe24cdef440dfa39fe42b57cd8939669afdf2f3182bcf69b7151b81c"}
Jul 28 17:13:17.085391 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:17.085220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerStarted","Data":"4ac591be521dc587d4ae6eb7aa7f7bd3b4c9f20cb25c9e5c8319a0c4f60f107c"}
Jul 28 17:13:17.085391 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:17.085263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfd7f79-2cb6-423d-ad0a-86aa70b53102","Type":"ContainerStarted","Data":"9169d9ccf8eca7ddda8b3936ca58f4e7cf5ff0b89336e3546a88ceede447878b"}
Jul 28 17:13:17.123272 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:17.123228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.103651363 podStartE2EDuration="2.123208545s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.557182ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:17.122017445 +0000 UTC m=+152.287653280" watchObservedRunningTime="2026-07-28 17:13:17.123208545 +0000 UTC m=+152.288844374"
Jul 28 17:13:18.978083 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.978045    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b554b58cf-6bk2z"]
Jul 28 17:13:18.980001 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.979985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:18.983181 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.983152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:13:18.983369 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.983350    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:13:18.983468 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.983413    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:13:18.983468 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.983429    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:13:18.983629 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.983614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:13:18.983729 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.983715    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:13:18.984855 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.984841    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:13:18.985046 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.985021    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hcgpx\""
Jul 28 17:13:18.989753 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.989738    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:13:18.995587 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:18.995570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b554b58cf-6bk2z"]
Jul 28 17:13:19.013560 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.013533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-service-ca\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.013680 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.013578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-serving-cert\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.013680 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.013598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-console-config\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.013787 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.013675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-oauth-config\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.013787 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.013712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-trusted-ca-bundle\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.013787 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.013770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-oauth-serving-cert\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.013913 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.013820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4xkh\" (UniqueName: \"kubernetes.io/projected/b01a57ad-404b-4510-a593-c8d58186d55a-kube-api-access-l4xkh\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.037996 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.037967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:19.097967 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.097932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:19.114717 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.114691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-serving-cert\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.114794 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.114722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-console-config\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.114794 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.114750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-oauth-config\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.114794 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.114764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-trusted-ca-bundle\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.114794 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.114790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-oauth-serving-cert\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.114984 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.114966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4xkh\" (UniqueName: \"kubernetes.io/projected/b01a57ad-404b-4510-a593-c8d58186d55a-kube-api-access-l4xkh\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.115041 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.115002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-service-ca\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.115637 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.115613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-console-config\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.115637 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.115625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-oauth-serving-cert\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.115788 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.115707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-service-ca\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.115788 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.115770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-trusted-ca-bundle\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.117190 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.117167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-oauth-config\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.117326 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.117287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-serving-cert\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.123484 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.123460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4xkh\" (UniqueName: \"kubernetes.io/projected/b01a57ad-404b-4510-a593-c8d58186d55a-kube-api-access-l4xkh\") pod \"console-7b554b58cf-6bk2z\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") " pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.290734 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.290642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:19.411581 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:19.411552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b554b58cf-6bk2z"]
Jul 28 17:13:19.413934 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:13:19.413904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01a57ad_404b_4510_a593_c8d58186d55a.slice/crio-faa69825e1c628f6992d97acc00c2267b1193c929d5e90d45f1202be76abd340 WatchSource:0}: Error finding container faa69825e1c628f6992d97acc00c2267b1193c929d5e90d45f1202be76abd340: Status 404 returned error can't find the container with id faa69825e1c628f6992d97acc00c2267b1193c929d5e90d45f1202be76abd340
Jul 28 17:13:20.101044 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:20.101007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b554b58cf-6bk2z" event={"ID":"b01a57ad-404b-4510-a593-c8d58186d55a","Type":"ContainerStarted","Data":"fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007"}
Jul 28 17:13:20.101562 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:20.101100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b554b58cf-6bk2z" event={"ID":"b01a57ad-404b-4510-a593-c8d58186d55a","Type":"ContainerStarted","Data":"faa69825e1c628f6992d97acc00c2267b1193c929d5e90d45f1202be76abd340"}
Jul 28 17:13:20.122377 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:20.122322    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b554b58cf-6bk2z" podStartSLOduration=2.122290738 podStartE2EDuration="2.122290738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:20.121103202 +0000 UTC m=+155.286739037" watchObservedRunningTime="2026-07-28 17:13:20.122290738 +0000 UTC m=+155.287926573"
Jul 28 17:13:20.441527 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:20.441430    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:25.171477 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:25.171441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:25.340433 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:25.340403    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:29.291033 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:29.291003    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:29.291554 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:29.291069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:29.295560 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:29.295533    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:30.142527 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:30.142499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:13:30.290568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:30.290523    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:30.396052 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:30.395974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:37.226099 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:37.226017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:37.362564 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:13:37.362528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:15.441689 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:15.441656    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:14:15.456594 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:15.456573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:14:16.314771 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:16.314732    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:14:18.742704 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:18.742670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:24.476077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.476040    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:24.576686 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.576642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:24.755583 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.755505    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bfcc6b4df-xgthd"]
Jul 28 17:14:24.761864 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.761833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:24.772247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.772214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfcc6b4df-xgthd"]
Jul 28 17:14:24.807805 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.807754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:24.909040 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.909009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:24.930165 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.930136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-trusted-ca-bundle\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:24.930293 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.930177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-oauth-serving-cert\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:24.930293 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.930219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-serving-cert\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:24.930293 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.930254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-oauth-config\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:24.930444 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.930338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-config\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:24.930444 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.930364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2nw\" (UniqueName: \"kubernetes.io/projected/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-kube-api-access-sz2nw\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:24.930444 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:24.930398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-service-ca\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.009198 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.009135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:25.031214 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.031189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-service-ca\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.031338 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.031239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-trusted-ca-bundle\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.031338 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.031275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-oauth-serving-cert\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.031464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.031336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-serving-cert\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.031464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.031375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-oauth-config\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.031464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.031413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-config\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.031464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.031438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2nw\" (UniqueName: \"kubernetes.io/projected/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-kube-api-access-sz2nw\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.032039 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.032012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-service-ca\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.032144 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.032055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-config\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.032144 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.032075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-oauth-serving-cert\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.032144 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.032133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-trusted-ca-bundle\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.034239 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.034220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-oauth-config\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.034343 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.034322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-serving-cert\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.040841 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.040823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2nw\" (UniqueName: \"kubernetes.io/projected/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-kube-api-access-sz2nw\") pod \"console-7bfcc6b4df-xgthd\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") " pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.070745 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.070719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:25.200721 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.200618    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfcc6b4df-xgthd"]
Jul 28 17:14:25.333433 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.333398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfcc6b4df-xgthd" event={"ID":"bfc809e2-5851-4d09-af5a-ff5c89eb4f11","Type":"ContainerStarted","Data":"1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b"}
Jul 28 17:14:25.333594 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.333452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfcc6b4df-xgthd" event={"ID":"bfc809e2-5851-4d09-af5a-ff5c89eb4f11","Type":"ContainerStarted","Data":"a144a52e1e7709cc18f7f668b61efa62aa37765abc6e5458edf801779621219b"}
Jul 28 17:14:25.354093 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.354041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bfcc6b4df-xgthd" podStartSLOduration=1.354026949 podStartE2EDuration="1.354026949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:25.352923059 +0000 UTC m=+220.518558909" watchObservedRunningTime="2026-07-28 17:14:25.354026949 +0000 UTC m=+220.519662782"
Jul 28 17:14:25.569774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.569695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:25.670799 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:25.670770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:30.650077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:30.650041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:35.071527 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.071496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:35.071932 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.071809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:35.076224 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.076198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:35.372591 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.372564    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:14:35.423135 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.423104    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b554b58cf-6bk2z"]
Jul 28 17:14:35.525548 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.525510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:35.640689 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.640606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:35.769117 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:35.769089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:36.580124 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:36.580069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:36.649246 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:36.649213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:50.275364 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:50.275328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:50.375814 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:14:50.375786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:00.442643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.442599    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b554b58cf-6bk2z" podUID="b01a57ad-404b-4510-a593-c8d58186d55a" containerName="console" containerID="cri-o://fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007" gracePeriod=15
Jul 28 17:15:00.683058 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.683035    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:15:00.812127 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-oauth-serving-cert\") pod \"b01a57ad-404b-4510-a593-c8d58186d55a\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") "
Jul 28 17:15:00.812127 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-serving-cert\") pod \"b01a57ad-404b-4510-a593-c8d58186d55a\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") "
Jul 28 17:15:00.812346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812161    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-service-ca\") pod \"b01a57ad-404b-4510-a593-c8d58186d55a\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") "
Jul 28 17:15:00.812346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812226    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-console-config\") pod \"b01a57ad-404b-4510-a593-c8d58186d55a\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") "
Jul 28 17:15:00.812346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-trusted-ca-bundle\") pod \"b01a57ad-404b-4510-a593-c8d58186d55a\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") "
Jul 28 17:15:00.812346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812323    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-oauth-config\") pod \"b01a57ad-404b-4510-a593-c8d58186d55a\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") "
Jul 28 17:15:00.812566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4xkh\" (UniqueName: \"kubernetes.io/projected/b01a57ad-404b-4510-a593-c8d58186d55a-kube-api-access-l4xkh\") pod \"b01a57ad-404b-4510-a593-c8d58186d55a\" (UID: \"b01a57ad-404b-4510-a593-c8d58186d55a\") "
Jul 28 17:15:00.812566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812463    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b01a57ad-404b-4510-a593-c8d58186d55a" (UID: "b01a57ad-404b-4510-a593-c8d58186d55a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:00.812672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812608    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-oauth-serving-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:15:00.812718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-console-config" (OuterVolumeSpecName: "console-config") pod "b01a57ad-404b-4510-a593-c8d58186d55a" (UID: "b01a57ad-404b-4510-a593-c8d58186d55a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:00.812718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-service-ca" (OuterVolumeSpecName: "service-ca") pod "b01a57ad-404b-4510-a593-c8d58186d55a" (UID: "b01a57ad-404b-4510-a593-c8d58186d55a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:00.812718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.812700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b01a57ad-404b-4510-a593-c8d58186d55a" (UID: "b01a57ad-404b-4510-a593-c8d58186d55a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:15:00.814465 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.814443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01a57ad-404b-4510-a593-c8d58186d55a-kube-api-access-l4xkh" (OuterVolumeSpecName: "kube-api-access-l4xkh") pod "b01a57ad-404b-4510-a593-c8d58186d55a" (UID: "b01a57ad-404b-4510-a593-c8d58186d55a"). InnerVolumeSpecName "kube-api-access-l4xkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:15:00.814569 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.814547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b01a57ad-404b-4510-a593-c8d58186d55a" (UID: "b01a57ad-404b-4510-a593-c8d58186d55a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:15:00.814569 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.814558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b01a57ad-404b-4510-a593-c8d58186d55a" (UID: "b01a57ad-404b-4510-a593-c8d58186d55a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:15:00.913549 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.913516    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-oauth-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:15:00.913549 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.913542    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4xkh\" (UniqueName: \"kubernetes.io/projected/b01a57ad-404b-4510-a593-c8d58186d55a-kube-api-access-l4xkh\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:15:00.913549 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.913551    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b01a57ad-404b-4510-a593-c8d58186d55a-console-serving-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:15:00.913736 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.913561    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:15:00.913736 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.913571    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-console-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:15:00.913736 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:00.913579    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01a57ad-404b-4510-a593-c8d58186d55a-trusted-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:15:01.455925 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.455893    2578 generic.go:356] "Generic (PLEG): container finished" podID="b01a57ad-404b-4510-a593-c8d58186d55a" containerID="fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007" exitCode=0
Jul 28 17:15:01.455925 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.455927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b554b58cf-6bk2z" event={"ID":"b01a57ad-404b-4510-a593-c8d58186d55a","Type":"ContainerDied","Data":"fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007"}
Jul 28 17:15:01.456367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.455935    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b554b58cf-6bk2z"
Jul 28 17:15:01.456367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.455962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b554b58cf-6bk2z" event={"ID":"b01a57ad-404b-4510-a593-c8d58186d55a","Type":"ContainerDied","Data":"faa69825e1c628f6992d97acc00c2267b1193c929d5e90d45f1202be76abd340"}
Jul 28 17:15:01.456367 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.455984    2578 scope.go:117] "RemoveContainer" containerID="fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007"
Jul 28 17:15:01.463781 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.463762    2578 scope.go:117] "RemoveContainer" containerID="fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007"
Jul 28 17:15:01.464012 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:15:01.463993    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007\": container with ID starting with fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007 not found: ID does not exist" containerID="fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007"
Jul 28 17:15:01.464072 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.464019    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007"} err="failed to get container status \"fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007\": rpc error: code = NotFound desc = could not find container \"fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007\": container with ID starting with fe7dac20dc867d1f563bcd0806d173dbba42ae2a40acd6e6f74c92260eed7007 not found: ID does not exist"
Jul 28 17:15:01.474478 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.474453    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b554b58cf-6bk2z"]
Jul 28 17:15:01.479321 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:01.479288    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b554b58cf-6bk2z"]
Jul 28 17:15:03.362245 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:03.362209    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01a57ad-404b-4510-a593-c8d58186d55a" path="/var/lib/kubelet/pods/b01a57ad-404b-4510-a593-c8d58186d55a/volumes"
Jul 28 17:15:45.181170 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:15:45.181136    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:16:50.053382 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:16:50.053350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:51.682145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:16:51.682097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:17.926676 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:17.926624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:18.023252 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:18.023220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:23.812103 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:23.812064    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.193737 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:24.193704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.355337 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:24.355291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.431518 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:24.431492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.532018 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:24.531943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.815653 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:24.815574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.913624 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:24.913593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.016092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:25.016062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.119863 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:25.119839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.251874 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:25.251835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.447340 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:25.447244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.545623 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:25.545592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.646006 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:25.645973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:49.094091 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:49.094054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:49.270985 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:17:49.270953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:20.664442 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.664368    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c8f4f955c-hw6l5"]
Jul 28 17:18:20.664917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.664625    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01a57ad-404b-4510-a593-c8d58186d55a" containerName="console"
Jul 28 17:18:20.664917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.664635    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01a57ad-404b-4510-a593-c8d58186d55a" containerName="console"
Jul 28 17:18:20.664917 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.664686    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01a57ad-404b-4510-a593-c8d58186d55a" containerName="console"
Jul 28 17:18:20.667843 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.667827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.687019 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.686995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8f4f955c-hw6l5"]
Jul 28 17:18:20.738511 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.738483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52aa805f-db0f-4931-920c-a318a64a4454-console-oauth-config\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.738610 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.738519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-service-ca\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.738610 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.738555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-trusted-ca-bundle\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.738610 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.738577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52aa805f-db0f-4931-920c-a318a64a4454-console-serving-cert\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.738610 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.738599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-oauth-serving-cert\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.738743 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.738660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-console-config\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.738743 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.738714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zbt\" (UniqueName: \"kubernetes.io/projected/52aa805f-db0f-4931-920c-a318a64a4454-kube-api-access-b7zbt\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.760286 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.760225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:20.839055 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-trusted-ca-bundle\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.839157 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52aa805f-db0f-4931-920c-a318a64a4454-console-serving-cert\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.839157 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-oauth-serving-cert\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.839157 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-console-config\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.839337 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zbt\" (UniqueName: \"kubernetes.io/projected/52aa805f-db0f-4931-920c-a318a64a4454-kube-api-access-b7zbt\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.839337 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52aa805f-db0f-4931-920c-a318a64a4454-console-oauth-config\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.839337 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-service-ca\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.839971 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-console-config\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.840064 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-oauth-serving-cert\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.840064 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.839960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-service-ca\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.840064 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.840018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52aa805f-db0f-4931-920c-a318a64a4454-trusted-ca-bundle\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.841467 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.841450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52aa805f-db0f-4931-920c-a318a64a4454-console-serving-cert\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.841601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.841582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52aa805f-db0f-4931-920c-a318a64a4454-console-oauth-config\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.900177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.900152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zbt\" (UniqueName: \"kubernetes.io/projected/52aa805f-db0f-4931-920c-a318a64a4454-kube-api-access-b7zbt\") pod \"console-c8f4f955c-hw6l5\" (UID: \"52aa805f-db0f-4931-920c-a318a64a4454\") " pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:20.958114 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.958053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:20.976021 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:20.975995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:21.055991 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:21.055920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:21.122193 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:21.122045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8f4f955c-hw6l5"]
Jul 28 17:18:21.124733 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:18:21.124708    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52aa805f_db0f_4931_920c_a318a64a4454.slice/crio-5faf969d6a81a4fd0c80a1d6a847b5b923ddcb92e2b6b1ca496d131675860e21 WatchSource:0}: Error finding container 5faf969d6a81a4fd0c80a1d6a847b5b923ddcb92e2b6b1ca496d131675860e21: Status 404 returned error can't find the container with id 5faf969d6a81a4fd0c80a1d6a847b5b923ddcb92e2b6b1ca496d131675860e21
Jul 28 17:18:21.146707 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:21.146685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8f4f955c-hw6l5" event={"ID":"52aa805f-db0f-4931-920c-a318a64a4454","Type":"ContainerStarted","Data":"5faf969d6a81a4fd0c80a1d6a847b5b923ddcb92e2b6b1ca496d131675860e21"}
Jul 28 17:18:21.475523 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:21.475488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:21.585082 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:21.585047    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:22.151574 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:22.151533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8f4f955c-hw6l5" event={"ID":"52aa805f-db0f-4931-920c-a318a64a4454","Type":"ContainerStarted","Data":"22c8ebafb2e02d7630439b3ae0dd79cc685555dc0f3cc23125797ef86ee5c867"}
Jul 28 17:18:22.176639 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:22.176590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c8f4f955c-hw6l5" podStartSLOduration=2.176574537 podStartE2EDuration="2.176574537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:22.175256434 +0000 UTC m=+457.340892262" watchObservedRunningTime="2026-07-28 17:18:22.176574537 +0000 UTC m=+457.342210371"
Jul 28 17:18:26.799200 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:26.799166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:26.905665 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:26.905637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:27.420269 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:27.420239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:30.976845 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:30.976798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:30.976845 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:30.976862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:30.981563 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:30.981542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:31.188243 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:31.188215    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c8f4f955c-hw6l5"
Jul 28 17:18:31.271040 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:31.270958    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bfcc6b4df-xgthd"]
Jul 28 17:18:31.486532 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:31.486501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:31.586525 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:31.586495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:43.569435 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:43.569389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:43.654093 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:43.654062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:43.861216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:43.861185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:43.962209 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:43.962176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:44.216358 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:44.216279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:44.314671 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:44.314632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:56.289379 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.289339    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bfcc6b4df-xgthd" podUID="bfc809e2-5851-4d09-af5a-ff5c89eb4f11" containerName="console" containerID="cri-o://1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b" gracePeriod=15
Jul 28 17:18:56.526261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.526238    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:18:56.614799 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.614773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-service-ca\") pod \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") "
Jul 28 17:18:56.614964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.614809    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-trusted-ca-bundle\") pod \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") "
Jul 28 17:18:56.614964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.614859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz2nw\" (UniqueName: \"kubernetes.io/projected/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-kube-api-access-sz2nw\") pod \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") "
Jul 28 17:18:56.614964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.614915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-oauth-config\") pod \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") "
Jul 28 17:18:56.614964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.614940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-config\") pod \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") "
Jul 28 17:18:56.615164 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.614970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-serving-cert\") pod \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") "
Jul 28 17:18:56.615164 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.615016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-oauth-serving-cert\") pod \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\" (UID: \"bfc809e2-5851-4d09-af5a-ff5c89eb4f11\") "
Jul 28 17:18:56.615164 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.615117    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-service-ca" (OuterVolumeSpecName: "service-ca") pod "bfc809e2-5851-4d09-af5a-ff5c89eb4f11" (UID: "bfc809e2-5851-4d09-af5a-ff5c89eb4f11"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:56.615329 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.615251    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-service-ca\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:18:56.615402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.615335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bfc809e2-5851-4d09-af5a-ff5c89eb4f11" (UID: "bfc809e2-5851-4d09-af5a-ff5c89eb4f11"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:56.615402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.615359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-config" (OuterVolumeSpecName: "console-config") pod "bfc809e2-5851-4d09-af5a-ff5c89eb4f11" (UID: "bfc809e2-5851-4d09-af5a-ff5c89eb4f11"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:56.615493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.615399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bfc809e2-5851-4d09-af5a-ff5c89eb4f11" (UID: "bfc809e2-5851-4d09-af5a-ff5c89eb4f11"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:56.616998 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.616961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bfc809e2-5851-4d09-af5a-ff5c89eb4f11" (UID: "bfc809e2-5851-4d09-af5a-ff5c89eb4f11"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:56.617102 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.617007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bfc809e2-5851-4d09-af5a-ff5c89eb4f11" (UID: "bfc809e2-5851-4d09-af5a-ff5c89eb4f11"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:56.617102 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.617038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-kube-api-access-sz2nw" (OuterVolumeSpecName: "kube-api-access-sz2nw") pod "bfc809e2-5851-4d09-af5a-ff5c89eb4f11" (UID: "bfc809e2-5851-4d09-af5a-ff5c89eb4f11"). InnerVolumeSpecName "kube-api-access-sz2nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:56.716230 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.716205    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-trusted-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:18:56.716230 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.716226    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz2nw\" (UniqueName: \"kubernetes.io/projected/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-kube-api-access-sz2nw\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:18:56.716413 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.716236    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-oauth-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:18:56.716413 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.716245    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:18:56.716413 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.716253    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-console-serving-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:18:56.716413 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:56.716262    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfc809e2-5851-4d09-af5a-ff5c89eb4f11-oauth-serving-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:18:57.280445 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.280415    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfc809e2-5851-4d09-af5a-ff5c89eb4f11" containerID="1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b" exitCode=0
Jul 28 17:18:57.280605 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.280457    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfcc6b4df-xgthd"
Jul 28 17:18:57.280605 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.280468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfcc6b4df-xgthd" event={"ID":"bfc809e2-5851-4d09-af5a-ff5c89eb4f11","Type":"ContainerDied","Data":"1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b"}
Jul 28 17:18:57.280605 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.280508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfcc6b4df-xgthd" event={"ID":"bfc809e2-5851-4d09-af5a-ff5c89eb4f11","Type":"ContainerDied","Data":"a144a52e1e7709cc18f7f668b61efa62aa37765abc6e5458edf801779621219b"}
Jul 28 17:18:57.280605 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.280529    2578 scope.go:117] "RemoveContainer" containerID="1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b"
Jul 28 17:18:57.288998 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.288982    2578 scope.go:117] "RemoveContainer" containerID="1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b"
Jul 28 17:18:57.289252 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:18:57.289231    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b\": container with ID starting with 1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b not found: ID does not exist" containerID="1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b"
Jul 28 17:18:57.289353 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.289263    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b"} err="failed to get container status \"1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b\": rpc error: code = NotFound desc = could not find container \"1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b\": container with ID starting with 1e8679dc08a74882cd6d73c971c25c038feee81daf8cf728fee22081862bb35b not found: ID does not exist"
Jul 28 17:18:57.301065 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.301047    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bfcc6b4df-xgthd"]
Jul 28 17:18:57.304018 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.303997    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bfcc6b4df-xgthd"]
Jul 28 17:18:57.362785 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:18:57.362759    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfc809e2-5851-4d09-af5a-ff5c89eb4f11" path="/var/lib/kubelet/pods/bfc809e2-5851-4d09-af5a-ff5c89eb4f11/volumes"
Jul 28 17:19:01.112214 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:19:01.112168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:01.377110 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.377075    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-mzsxc"]
Jul 28 17:20:01.377637 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.377476    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc809e2-5851-4d09-af5a-ff5c89eb4f11" containerName="console"
Jul 28 17:20:01.377637 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.377493    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc809e2-5851-4d09-af5a-ff5c89eb4f11" containerName="console"
Jul 28 17:20:01.377637 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.377638    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc809e2-5851-4d09-af5a-ff5c89eb4f11" containerName="console"
Jul 28 17:20:01.381550 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.381531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.384793 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.384763    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-qcklz\""
Jul 28 17:20:01.384887 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.384801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:20:01.386208 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.386189    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:20:01.386327 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.386192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:20:01.392624 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.392597    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-mzsxc"]
Jul 28 17:20:01.424915 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.424873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15953d4a-d6af-4302-ab0a-485dad92d135-tls-certs\") pod \"model-serving-api-7b954ffc95-mzsxc\" (UID: \"15953d4a-d6af-4302-ab0a-485dad92d135\") " pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.424915 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.424916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wmd\" (UniqueName: \"kubernetes.io/projected/15953d4a-d6af-4302-ab0a-485dad92d135-kube-api-access-x9wmd\") pod \"model-serving-api-7b954ffc95-mzsxc\" (UID: \"15953d4a-d6af-4302-ab0a-485dad92d135\") " pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.438692 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.438666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:01.514049 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.514018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:01.526181 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.526157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15953d4a-d6af-4302-ab0a-485dad92d135-tls-certs\") pod \"model-serving-api-7b954ffc95-mzsxc\" (UID: \"15953d4a-d6af-4302-ab0a-485dad92d135\") " pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.526263 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.526189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wmd\" (UniqueName: \"kubernetes.io/projected/15953d4a-d6af-4302-ab0a-485dad92d135-kube-api-access-x9wmd\") pod \"model-serving-api-7b954ffc95-mzsxc\" (UID: \"15953d4a-d6af-4302-ab0a-485dad92d135\") " pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.528579 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.528559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15953d4a-d6af-4302-ab0a-485dad92d135-tls-certs\") pod \"model-serving-api-7b954ffc95-mzsxc\" (UID: \"15953d4a-d6af-4302-ab0a-485dad92d135\") " pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.535490 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.535471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wmd\" (UniqueName: \"kubernetes.io/projected/15953d4a-d6af-4302-ab0a-485dad92d135-kube-api-access-x9wmd\") pod \"model-serving-api-7b954ffc95-mzsxc\" (UID: \"15953d4a-d6af-4302-ab0a-485dad92d135\") " pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.630343 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.630249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:01.691785 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.691757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:01.816573 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.816550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-mzsxc"]
Jul 28 17:20:01.818982 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:20:01.818959    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15953d4a_d6af_4302_ab0a_485dad92d135.slice/crio-33f88c4e86b143b00fb547ba247a77cab49c8e57aae48869fab2189095df1b1f WatchSource:0}: Error finding container 33f88c4e86b143b00fb547ba247a77cab49c8e57aae48869fab2189095df1b1f: Status 404 returned error can't find the container with id 33f88c4e86b143b00fb547ba247a77cab49c8e57aae48869fab2189095df1b1f
Jul 28 17:20:01.821216 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:01.821200    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:20:02.013744 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:02.013660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:02.115078 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:02.115051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:02.513169 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:02.513139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-mzsxc" event={"ID":"15953d4a-d6af-4302-ab0a-485dad92d135","Type":"ContainerStarted","Data":"33f88c4e86b143b00fb547ba247a77cab49c8e57aae48869fab2189095df1b1f"}
Jul 28 17:20:04.521461 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:04.521418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-mzsxc" event={"ID":"15953d4a-d6af-4302-ab0a-485dad92d135","Type":"ContainerStarted","Data":"34354ba0ae8d2f61d3c266168f1b21ef4f8c1e5078537b1653dccb3fee145477"}
Jul 28 17:20:04.521887 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:04.521573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:04.541369 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:04.541323    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-mzsxc" podStartSLOduration=0.99664291 podStartE2EDuration="3.541295401s" totalImagesPullingTime="2.544652491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:04.540827622 +0000 UTC m=+559.706463455" watchObservedRunningTime="2026-07-28 17:20:04.541295401 +0000 UTC m=+559.706931235"
Jul 28 17:20:15.529003 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:15.528976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-mzsxc"
Jul 28 17:20:17.396395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.396360    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-c64q5"]
Jul 28 17:20:17.404235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.404210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.407594 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.407566    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:20:17.407720 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.407676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9w9f9\""
Jul 28 17:20:17.411282 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.411261    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-c64q5"]
Jul 28 17:20:17.554084 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.554039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cv2\" (UniqueName: \"kubernetes.io/projected/463af8a3-0241-4563-a154-ac7b83beb165-kube-api-access-n5cv2\") pod \"s3-init-c64q5\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") " pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.554244 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.554142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463af8a3-0241-4563-a154-ac7b83beb165-model-cache\") pod \"s3-init-c64q5\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") " pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.654697 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.654587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cv2\" (UniqueName: \"kubernetes.io/projected/463af8a3-0241-4563-a154-ac7b83beb165-kube-api-access-n5cv2\") pod \"s3-init-c64q5\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") " pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.654697 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.654689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463af8a3-0241-4563-a154-ac7b83beb165-model-cache\") pod \"s3-init-c64q5\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") " pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.655062 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.655041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463af8a3-0241-4563-a154-ac7b83beb165-model-cache\") pod \"s3-init-c64q5\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") " pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.664927 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.664898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cv2\" (UniqueName: \"kubernetes.io/projected/463af8a3-0241-4563-a154-ac7b83beb165-kube-api-access-n5cv2\") pod \"s3-init-c64q5\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") " pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.721461 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.721439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c64q5"
Jul 28 17:20:17.844241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:17.844192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-c64q5"]
Jul 28 17:20:17.847349 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:20:17.847325    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod463af8a3_0241_4563_a154_ac7b83beb165.slice/crio-600da2321b5836fc6824ce951bc2109943dd9ce193177fb56286de85e9751b58 WatchSource:0}: Error finding container 600da2321b5836fc6824ce951bc2109943dd9ce193177fb56286de85e9751b58: Status 404 returned error can't find the container with id 600da2321b5836fc6824ce951bc2109943dd9ce193177fb56286de85e9751b58
Jul 28 17:20:18.573483 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:18.573442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c64q5" event={"ID":"463af8a3-0241-4563-a154-ac7b83beb165","Type":"ContainerStarted","Data":"600da2321b5836fc6824ce951bc2109943dd9ce193177fb56286de85e9751b58"}
Jul 28 17:20:21.588205 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:21.588174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c64q5" event={"ID":"463af8a3-0241-4563-a154-ac7b83beb165","Type":"ContainerStarted","Data":"9bd962e2ef60abfcb3b4409a67b0df93d08652693aa83c156e4441ad628f9a10"}
Jul 28 17:20:28.613482 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:28.613451    2578 generic.go:356] "Generic (PLEG): container finished" podID="463af8a3-0241-4563-a154-ac7b83beb165" containerID="9bd962e2ef60abfcb3b4409a67b0df93d08652693aa83c156e4441ad628f9a10" exitCode=0
Jul 28 17:20:28.613844 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:28.613520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c64q5" event={"ID":"463af8a3-0241-4563-a154-ac7b83beb165","Type":"ContainerDied","Data":"9bd962e2ef60abfcb3b4409a67b0df93d08652693aa83c156e4441ad628f9a10"}
Jul 28 17:20:33.637146 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:33.637110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c64q5" event={"ID":"463af8a3-0241-4563-a154-ac7b83beb165","Type":"ContainerStarted","Data":"46dc5108cde604049a5104f921be74612baa07e5e87383a7aa518757e3b1bfdd"}
Jul 28 17:20:33.656099 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:20:33.656054    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-c64q5" podStartSLOduration=1.866479384 podStartE2EDuration="16.65603902s" totalImagesPullingTime="7.985638006s" totalInitContainerRuntime="6.80392163s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:33.655077595 +0000 UTC m=+588.820713429" watchObservedRunningTime="2026-07-28 17:20:33.65603902 +0000 UTC m=+588.821674853"
Jul 28 17:21:11.817053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:11.816978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:38.875725 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:38.875695    2578 generic.go:356] "Generic (PLEG): container finished" podID="463af8a3-0241-4563-a154-ac7b83beb165" containerID="46dc5108cde604049a5104f921be74612baa07e5e87383a7aa518757e3b1bfdd" exitCode=0
Jul 28 17:21:38.876130 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:38.875773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c64q5" event={"ID":"463af8a3-0241-4563-a154-ac7b83beb165","Type":"ContainerDied","Data":"46dc5108cde604049a5104f921be74612baa07e5e87383a7aa518757e3b1bfdd"}
Jul 28 17:21:39.999629 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:39.999598    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c64q5"
Jul 28 17:21:40.119656 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.119624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5cv2\" (UniqueName: \"kubernetes.io/projected/463af8a3-0241-4563-a154-ac7b83beb165-kube-api-access-n5cv2\") pod \"463af8a3-0241-4563-a154-ac7b83beb165\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") "
Jul 28 17:21:40.119809 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.119679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463af8a3-0241-4563-a154-ac7b83beb165-model-cache\") pod \"463af8a3-0241-4563-a154-ac7b83beb165\" (UID: \"463af8a3-0241-4563-a154-ac7b83beb165\") "
Jul 28 17:21:40.121851 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.121829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463af8a3-0241-4563-a154-ac7b83beb165-kube-api-access-n5cv2" (OuterVolumeSpecName: "kube-api-access-n5cv2") pod "463af8a3-0241-4563-a154-ac7b83beb165" (UID: "463af8a3-0241-4563-a154-ac7b83beb165"). InnerVolumeSpecName "kube-api-access-n5cv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:21:40.138549 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.138492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463af8a3-0241-4563-a154-ac7b83beb165-model-cache" (OuterVolumeSpecName: "model-cache") pod "463af8a3-0241-4563-a154-ac7b83beb165" (UID: "463af8a3-0241-4563-a154-ac7b83beb165"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:21:40.220405 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.220379    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463af8a3-0241-4563-a154-ac7b83beb165-model-cache\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:21:40.220405 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.220403    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5cv2\" (UniqueName: \"kubernetes.io/projected/463af8a3-0241-4563-a154-ac7b83beb165-kube-api-access-n5cv2\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:21:40.883839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.883744    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-c64q5"
Jul 28 17:21:40.883839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.883759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-c64q5" event={"ID":"463af8a3-0241-4563-a154-ac7b83beb165","Type":"ContainerDied","Data":"600da2321b5836fc6824ce951bc2109943dd9ce193177fb56286de85e9751b58"}
Jul 28 17:21:40.883839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:40.883797    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="600da2321b5836fc6824ce951bc2109943dd9ce193177fb56286de85e9751b58"
Jul 28 17:21:41.245363 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:41.245267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:41.345712 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:41.345680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:53.067753 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.067715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:53.175311 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.175275    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"]
Jul 28 17:21:53.175563 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.175550    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463af8a3-0241-4563-a154-ac7b83beb165" containerName="s3-init"
Jul 28 17:21:53.175563 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.175564    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="463af8a3-0241-4563-a154-ac7b83beb165" containerName="s3-init"
Jul 28 17:21:53.175644 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.175580    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463af8a3-0241-4563-a154-ac7b83beb165" containerName="download-hf-model"
Jul 28 17:21:53.175644 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.175586    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="463af8a3-0241-4563-a154-ac7b83beb165" containerName="download-hf-model"
Jul 28 17:21:53.175644 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.175632    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="463af8a3-0241-4563-a154-ac7b83beb165" containerName="s3-init"
Jul 28 17:21:53.178693 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.178678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.181829 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.181794    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2ae79-kube-rbac-proxy-sar-config\""
Jul 28 17:21:53.181955 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.181830    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:21:53.182015 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.181970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:21:53.182111 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.182093    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w2njn\""
Jul 28 17:21:53.182309 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.182280    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:21:53.183503 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.183490    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2ae79-predictor-serving-cert\""
Jul 28 17:21:53.183565 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.183510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:21:53.192032 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.192011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"]
Jul 28 17:21:53.222899 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.222871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68fl\" (UniqueName: \"kubernetes.io/projected/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-api-access-n68fl\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.223020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.222906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.223020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.222927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.223115 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.223018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.223115 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.223052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0781fd07-5119-4b59-ae2e-225c9222aea1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.323817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.323751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68fl\" (UniqueName: \"kubernetes.io/projected/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-api-access-n68fl\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.323817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.323787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.323956 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.323815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.323956 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.323869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.323956 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.323901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0781fd07-5119-4b59-ae2e-225c9222aea1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.324088 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:21:53.323989    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-2ae79-predictor-serving-cert" not found
Jul 28 17:21:53.324088 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:21:53.324051    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls podName:0781fd07-5119-4b59-ae2e-225c9222aea1 nodeName:}" failed. No retries permitted until 2026-07-28 17:21:53.824030949 +0000 UTC m=+668.989666775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls") pod "isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" (UID: "0781fd07-5119-4b59-ae2e-225c9222aea1") : secret "isvc-raw-sklearn-batcher-2ae79-predictor-serving-cert" not found
Jul 28 17:21:53.324291 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.324272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0781fd07-5119-4b59-ae2e-225c9222aea1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.324603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.324576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.324603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.324595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.332958 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.332939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68fl\" (UniqueName: \"kubernetes.io/projected/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-api-access-n68fl\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.827421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.827380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:53.829774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:53.829746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:54.088560 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:54.088460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:21:54.215731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:54.215703    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"]
Jul 28 17:21:54.217691 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:21:54.217660    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0781fd07_5119_4b59_ae2e_225c9222aea1.slice/crio-548245979b3f75e36b0b29e4f7111d91be7ff5cd336c8ed784ab1e2cd34ffa07 WatchSource:0}: Error finding container 548245979b3f75e36b0b29e4f7111d91be7ff5cd336c8ed784ab1e2cd34ffa07: Status 404 returned error can't find the container with id 548245979b3f75e36b0b29e4f7111d91be7ff5cd336c8ed784ab1e2cd34ffa07
Jul 28 17:21:54.378744 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:54.378659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:54.940122 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:54.940087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerStarted","Data":"967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d"}
Jul 28 17:21:54.940285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:54.940147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerStarted","Data":"548245979b3f75e36b0b29e4f7111d91be7ff5cd336c8ed784ab1e2cd34ffa07"}
Jul 28 17:21:58.955481 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:58.955446    2578 generic.go:356] "Generic (PLEG): container finished" podID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerID="967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d" exitCode=0
Jul 28 17:21:58.955881 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:21:58.955510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerDied","Data":"967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d"}
Jul 28 17:22:06.470572 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:06.470521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:06.677487 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:06.677455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:12.020816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:12.020740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerStarted","Data":"6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2"}
Jul 28 17:22:14.031031 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:14.031000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerStarted","Data":"c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8"}
Jul 28 17:22:16.040414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:16.040331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerStarted","Data":"6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6"}
Jul 28 17:22:16.040760 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:16.040460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:22:16.065675 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:16.065626    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podStartSLOduration=2.597959665 podStartE2EDuration="23.065610782s" totalImagesPullingTime="16.686872127s" totalInitContainerRuntime="3.78077899s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:16.063054632 +0000 UTC m=+691.228690466" watchObservedRunningTime="2026-07-28 17:22:16.065610782 +0000 UTC m=+691.231246616"
Jul 28 17:22:17.044584 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:17.044550    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:22:17.045039 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:17.044626    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:22:17.045855 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:17.045822    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:22:17.046872 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:17.046850    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:18.048533 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:18.048489    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:22:18.048926 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:18.048888    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:23.052936 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:23.052907    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:22:23.053599 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:23.053564    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:22:23.053883 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:23.053855    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:33.053745 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:33.053703    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:22:33.054212 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:33.054130    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:43.054131 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:43.054084    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:22:43.054681 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:43.054527    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:53.053905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:53.053860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:22:53.054343 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:22:53.054316    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:03.054261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:03.054221    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:23:03.054773 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:03.054749    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:13.053539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:13.053492    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:23:13.054015 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:13.053959    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:19.742845 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:19.742813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:23.053970 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:23.053924    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:23:23.054375 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:23.054344    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:30.358468 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:30.358436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:23:30.359176 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:30.358589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:23:38.266865 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.266833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:38.318631 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.318601    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"]
Jul 28 17:23:38.318930 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.318899    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" containerID="cri-o://6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2" gracePeriod=30
Jul 28 17:23:38.319088 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.318941    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" containerID="cri-o://6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6" gracePeriod=30
Jul 28 17:23:38.319088 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.319002    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy" containerID="cri-o://c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8" gracePeriod=30
Jul 28 17:23:38.322008 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.321985    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"]
Jul 28 17:23:38.325694 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.325678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.328555 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.328535    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-02b1d-predictor-serving-cert\""
Jul 28 17:23:38.328970 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.328952    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-02b1d-kube-rbac-proxy-sar-config\""
Jul 28 17:23:38.341272 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.341250    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"]
Jul 28 17:23:38.352695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.352664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:38.408579 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.408555    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"]
Jul 28 17:23:38.411792 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.411777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.414564 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.414547    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-02b1d-predictor-serving-cert\""
Jul 28 17:23:38.414564 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.414560    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-02b1d-kube-rbac-proxy-sar-config\""
Jul 28 17:23:38.422039 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.422018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"]
Jul 28 17:23:38.430826 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.430790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:38.473437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.473413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbxr\" (UniqueName: \"kubernetes.io/projected/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-api-access-8sbxr\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.473580 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.473446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/097f5690-8e64-41a2-ba73-01fc79aeb386-proxy-tls\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.473580 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.473527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.473580 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.473557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/097f5690-8e64-41a2-ba73-01fc79aeb386-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.473753 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.473637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-cabundle-cert\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.528643 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.528569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:38.574761 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.574735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.574898 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.574787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.574898 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.574842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/097f5690-8e64-41a2-ba73-01fc79aeb386-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.574898 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.574871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc41b91-8d03-4da5-8999-e07a01af5cfa-proxy-tls\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.575071 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.574896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t276\" (UniqueName: \"kubernetes.io/projected/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-api-access-5t276\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.575071 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.574947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-cabundle-cert\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.575071 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.574995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbxr\" (UniqueName: \"kubernetes.io/projected/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-api-access-8sbxr\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.575071 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.575027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/097f5690-8e64-41a2-ba73-01fc79aeb386-proxy-tls\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.575071 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.575057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.575283 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.575085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.575283 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.575242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/097f5690-8e64-41a2-ba73-01fc79aeb386-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.575503 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.575484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.575557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.575545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-cabundle-cert\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.577445 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.577428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/097f5690-8e64-41a2-ba73-01fc79aeb386-proxy-tls\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.584255 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.584233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbxr\" (UniqueName: \"kubernetes.io/projected/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-api-access-8sbxr\") pod \"isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.642869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.642843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:38.671768 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.671731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:38.675648 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.675627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.675744 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.675675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc41b91-8d03-4da5-8999-e07a01af5cfa-proxy-tls\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.675744 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.675694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t276\" (UniqueName: \"kubernetes.io/projected/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-api-access-5t276\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.675824 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.675759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.675824 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.675786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.676244 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.676207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.676461 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.676439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.676596 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.676579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.677984 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.677966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc41b91-8d03-4da5-8999-e07a01af5cfa-proxy-tls\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.685265 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.685246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t276\" (UniqueName: \"kubernetes.io/projected/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-api-access-5t276\") pod \"isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.722344 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.722284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:23:38.778937 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.778861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"]
Jul 28 17:23:38.780476 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:23:38.780449    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097f5690_8e64_41a2_ba73_01fc79aeb386.slice/crio-0e537da6c6854c544d7570f448a84b660345b8e4d7ceb3b21308015f79568e9a WatchSource:0}: Error finding container 0e537da6c6854c544d7570f448a84b660345b8e4d7ceb3b21308015f79568e9a: Status 404 returned error can't find the container with id 0e537da6c6854c544d7570f448a84b660345b8e4d7ceb3b21308015f79568e9a
Jul 28 17:23:38.861238 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:38.861215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"]
Jul 28 17:23:38.863130 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:23:38.863098    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc41b91_8d03_4da5_8999_e07a01af5cfa.slice/crio-4f4f2e878c8b2a6d6f8b37ce0b2e9cc3d5aa934733561bf64bc84dd05a9b2cd7 WatchSource:0}: Error finding container 4f4f2e878c8b2a6d6f8b37ce0b2e9cc3d5aa934733561bf64bc84dd05a9b2cd7: Status 404 returned error can't find the container with id 4f4f2e878c8b2a6d6f8b37ce0b2e9cc3d5aa934733561bf64bc84dd05a9b2cd7
Jul 28 17:23:39.364890 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:39.364852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerStarted","Data":"ff40ddf287f0e97fd57b69c0c273e699ed6d3a637dc117ac4564349fe7355ce6"}
Jul 28 17:23:39.365475 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:39.365443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerStarted","Data":"0e537da6c6854c544d7570f448a84b660345b8e4d7ceb3b21308015f79568e9a"}
Jul 28 17:23:39.365593 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:39.365501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerStarted","Data":"d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e"}
Jul 28 17:23:39.365593 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:39.365536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerStarted","Data":"4f4f2e878c8b2a6d6f8b37ce0b2e9cc3d5aa934733561bf64bc84dd05a9b2cd7"}
Jul 28 17:23:39.365593 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:39.365567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerDied","Data":"c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8"}
Jul 28 17:23:39.365700 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:39.365267    2578 generic.go:356] "Generic (PLEG): container finished" podID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerID="c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8" exitCode=2
Jul 28 17:23:40.358478 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:40.358427    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:23:40.358741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:40.358716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:43.048747 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:43.048657    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 17:23:43.381928 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:43.381888    2578 generic.go:356] "Generic (PLEG): container finished" podID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerID="6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2" exitCode=0
Jul 28 17:23:43.382099 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:43.381956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerDied","Data":"6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2"}
Jul 28 17:23:43.383263 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:43.383243    2578 generic.go:356] "Generic (PLEG): container finished" podID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerID="ff40ddf287f0e97fd57b69c0c273e699ed6d3a637dc117ac4564349fe7355ce6" exitCode=0
Jul 28 17:23:43.383395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:43.383331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerDied","Data":"ff40ddf287f0e97fd57b69c0c273e699ed6d3a637dc117ac4564349fe7355ce6"}
Jul 28 17:23:43.384802 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:43.384783    2578 generic.go:356] "Generic (PLEG): container finished" podID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerID="d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e" exitCode=0
Jul 28 17:23:43.384888 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:43.384854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerDied","Data":"d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e"}
Jul 28 17:23:44.391129 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:44.391084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerStarted","Data":"58ba6bfa0621fc0ddc83f7f5a709176029475631ed873e30749ed66e445ab394"}
Jul 28 17:23:44.391616 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:44.391170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerStarted","Data":"8330602f7a79ed17da8cb2523a95c6d7ff44a03875f9a1c0f9699fe0e726cf71"}
Jul 28 17:23:44.391616 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:44.391409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:44.415226 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:44.415165    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podStartSLOduration=2.854157849 podStartE2EDuration="6.415142052s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.560984203s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:44.412039332 +0000 UTC m=+779.577675165" watchObservedRunningTime="2026-07-28 17:23:44.415142052 +0000 UTC m=+779.580777887"
Jul 28 17:23:45.397100 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:45.397072    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:45.398880 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:45.398843    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:23:46.405510 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:46.405464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:23:48.048704 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:48.048662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 17:23:50.358204 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:50.358148    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:23:50.358683 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:50.358542    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:51.411884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:51.411853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:23:51.412537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:51.412505    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:23:53.048687 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:53.048651    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 17:23:53.049130 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:53.048796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:23:58.049318 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:23:58.049251    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 17:24:00.358187 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:00.358143    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 17:24:00.358670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:00.358289    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:24:00.358670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:00.358600    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:00.358785 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:00.358713    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:24:01.413240 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:01.413200    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:24:01.474377 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:01.474348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerStarted","Data":"d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf"}
Jul 28 17:24:01.474498 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:01.474411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerStarted","Data":"4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b"}
Jul 28 17:24:01.474670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:01.474656    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:24:01.500179 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:01.500132    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podStartSLOduration=2.353087252 podStartE2EDuration="23.500118605s" totalImagesPullingTime="17.589486512s" totalInitContainerRuntime="3.557544841s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:01.499517894 +0000 UTC m=+796.665153730" watchObservedRunningTime="2026-07-28 17:24:01.500118605 +0000 UTC m=+796.665754439"
Jul 28 17:24:02.478015 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:02.477986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:24:02.479277 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:02.479249    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:24:03.049395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:03.049351    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 17:24:03.481340 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:03.481278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:24:08.049354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:08.049265    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 17:24:08.485985 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:08.485957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:24:08.486448 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:08.486424    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:24:09.026955 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.026931    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:24:09.148466 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0781fd07-5119-4b59-ae2e-225c9222aea1-kserve-provision-location\") pod \"0781fd07-5119-4b59-ae2e-225c9222aea1\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") "
Jul 28 17:24:09.148466 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n68fl\" (UniqueName: \"kubernetes.io/projected/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-api-access-n68fl\") pod \"0781fd07-5119-4b59-ae2e-225c9222aea1\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") "
Jul 28 17:24:09.148466 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls\") pod \"0781fd07-5119-4b59-ae2e-225c9222aea1\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") "
Jul 28 17:24:09.148978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148510    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-cabundle-cert\") pod \"0781fd07-5119-4b59-ae2e-225c9222aea1\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") "
Jul 28 17:24:09.148978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-rbac-proxy-sar-config\") pod \"0781fd07-5119-4b59-ae2e-225c9222aea1\" (UID: \"0781fd07-5119-4b59-ae2e-225c9222aea1\") "
Jul 28 17:24:09.148978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0781fd07-5119-4b59-ae2e-225c9222aea1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0781fd07-5119-4b59-ae2e-225c9222aea1" (UID: "0781fd07-5119-4b59-ae2e-225c9222aea1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:09.148978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0781fd07-5119-4b59-ae2e-225c9222aea1" (UID: "0781fd07-5119-4b59-ae2e-225c9222aea1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:09.148978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.148951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0781fd07-5119-4b59-ae2e-225c9222aea1" (UID: "0781fd07-5119-4b59-ae2e-225c9222aea1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:09.150637 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.150616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-api-access-n68fl" (OuterVolumeSpecName: "kube-api-access-n68fl") pod "0781fd07-5119-4b59-ae2e-225c9222aea1" (UID: "0781fd07-5119-4b59-ae2e-225c9222aea1"). InnerVolumeSpecName "kube-api-access-n68fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:09.150717 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.150692    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0781fd07-5119-4b59-ae2e-225c9222aea1" (UID: "0781fd07-5119-4b59-ae2e-225c9222aea1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:09.249248 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.249218    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0781fd07-5119-4b59-ae2e-225c9222aea1-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.249248 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.249248    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68fl\" (UniqueName: \"kubernetes.io/projected/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-api-access-n68fl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.249449 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.249263    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0781fd07-5119-4b59-ae2e-225c9222aea1-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.249449 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.249279    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.249449 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.249290    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0781fd07-5119-4b59-ae2e-225c9222aea1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.506881 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.506799    2578 generic.go:356] "Generic (PLEG): container finished" podID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerID="6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6" exitCode=0
Jul 28 17:24:09.506881 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.506847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerDied","Data":"6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6"}
Jul 28 17:24:09.506881 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.506871    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"
Jul 28 17:24:09.507164 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.506908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7" event={"ID":"0781fd07-5119-4b59-ae2e-225c9222aea1","Type":"ContainerDied","Data":"548245979b3f75e36b0b29e4f7111d91be7ff5cd336c8ed784ab1e2cd34ffa07"}
Jul 28 17:24:09.507164 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.506943    2578 scope.go:117] "RemoveContainer" containerID="6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6"
Jul 28 17:24:09.514925 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.514905    2578 scope.go:117] "RemoveContainer" containerID="c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8"
Jul 28 17:24:09.521741 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.521717    2578 scope.go:117] "RemoveContainer" containerID="6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2"
Jul 28 17:24:09.529024 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.529002    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"]
Jul 28 17:24:09.530048 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.529966    2578 scope.go:117] "RemoveContainer" containerID="967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d"
Jul 28 17:24:09.532117 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.532098    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ae79-predictor-9c96f899b-dh5c7"]
Jul 28 17:24:09.536993 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.536979    2578 scope.go:117] "RemoveContainer" containerID="6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6"
Jul 28 17:24:09.537232 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:24:09.537212    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6\": container with ID starting with 6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6 not found: ID does not exist" containerID="6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6"
Jul 28 17:24:09.537323 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.537244    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6"} err="failed to get container status \"6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6\": rpc error: code = NotFound desc = could not find container \"6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6\": container with ID starting with 6ef1697fe3501dc538541ae1987a4ac869b3be2ff650aa6fb4a1a0284351a9f6 not found: ID does not exist"
Jul 28 17:24:09.537323 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.537267    2578 scope.go:117] "RemoveContainer" containerID="c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8"
Jul 28 17:24:09.537532 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:24:09.537511    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8\": container with ID starting with c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8 not found: ID does not exist" containerID="c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8"
Jul 28 17:24:09.537623 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.537535    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8"} err="failed to get container status \"c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8\": rpc error: code = NotFound desc = could not find container \"c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8\": container with ID starting with c779b4de3e8e9974fc7939dd54385929ee92631031eae6a6dfb75191b1d213a8 not found: ID does not exist"
Jul 28 17:24:09.537623 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.537550    2578 scope.go:117] "RemoveContainer" containerID="6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2"
Jul 28 17:24:09.537771 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:24:09.537756    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2\": container with ID starting with 6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2 not found: ID does not exist" containerID="6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2"
Jul 28 17:24:09.537817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.537776    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2"} err="failed to get container status \"6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2\": rpc error: code = NotFound desc = could not find container \"6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2\": container with ID starting with 6344f7c8ff92904111056805ec7a4348fc1249f0d782b66ec994020bbca158f2 not found: ID does not exist"
Jul 28 17:24:09.537817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.537789    2578 scope.go:117] "RemoveContainer" containerID="967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d"
Jul 28 17:24:09.537975 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:24:09.537960    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d\": container with ID starting with 967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d not found: ID does not exist" containerID="967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d"
Jul 28 17:24:09.538017 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:09.537982    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d"} err="failed to get container status \"967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d\": rpc error: code = NotFound desc = could not find container \"967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d\": container with ID starting with 967be6308124f87cd6b26e64dc10aea21bbb8ddf2c67a1dc184bb8cb4cee531d not found: ID does not exist"
Jul 28 17:24:11.362204 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:11.362169    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" path="/var/lib/kubelet/pods/0781fd07-5119-4b59-ae2e-225c9222aea1/volumes"
Jul 28 17:24:11.413153 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:11.413118    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:24:18.486708 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:18.486665    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:24:21.412805 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:21.412764    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:24:28.487275 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:28.487237    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:24:31.413276 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:31.413232    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:24:38.486914 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:38.486874    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:24:41.412976 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:41.412925    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:24:48.487348 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:48.487284    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:24:51.413047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:51.413010    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:24:58.487222 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:24:58.487185    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:25:01.413113 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:01.413087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:25:08.486899 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:08.486869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:25:18.343293 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343259    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"]
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343600    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343612    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343619    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343625    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343634    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="storage-initializer"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343640    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="storage-initializer"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343646    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343651    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343691    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kserve-container"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343700    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="agent"
Jul 28 17:25:18.343759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.343708    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0781fd07-5119-4b59-ae2e-225c9222aea1" containerName="kube-rbac-proxy"
Jul 28 17:25:18.346322 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.346291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:18.349290 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.349267    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-02b1d-kube-rbac-proxy-sar-config\""
Jul 28 17:25:18.349532 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.349515    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-02b1d-serving-cert\""
Jul 28 17:25:18.355021 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.355000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"]
Jul 28 17:25:18.375052 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.375032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:18.422182 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.422155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls\") pod \"model-chainer-raw-02b1d-68ddfd4db5-tp8p9\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") " pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:18.422285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.422193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e209cfbb-7730-41c8-af11-d67a21eb9f63-openshift-service-ca-bundle\") pod \"model-chainer-raw-02b1d-68ddfd4db5-tp8p9\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") " pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:18.523382 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.523353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls\") pod \"model-chainer-raw-02b1d-68ddfd4db5-tp8p9\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") " pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:18.523537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.523389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e209cfbb-7730-41c8-af11-d67a21eb9f63-openshift-service-ca-bundle\") pod \"model-chainer-raw-02b1d-68ddfd4db5-tp8p9\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") " pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:18.523537 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:18.523508    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-02b1d-serving-cert: secret "model-chainer-raw-02b1d-serving-cert" not found
Jul 28 17:25:18.523656 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:18.523580    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls podName:e209cfbb-7730-41c8-af11-d67a21eb9f63 nodeName:}" failed. No retries permitted until 2026-07-28 17:25:19.023558927 +0000 UTC m=+874.189194739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls") pod "model-chainer-raw-02b1d-68ddfd4db5-tp8p9" (UID: "e209cfbb-7730-41c8-af11-d67a21eb9f63") : secret "model-chainer-raw-02b1d-serving-cert" not found
Jul 28 17:25:18.523973 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.523954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e209cfbb-7730-41c8-af11-d67a21eb9f63-openshift-service-ca-bundle\") pod \"model-chainer-raw-02b1d-68ddfd4db5-tp8p9\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") " pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:18.651103 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:18.650993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:19.026402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:19.026319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls\") pod \"model-chainer-raw-02b1d-68ddfd4db5-tp8p9\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") " pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:19.028670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:19.028651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls\") pod \"model-chainer-raw-02b1d-68ddfd4db5-tp8p9\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") " pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:19.257049 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:19.257008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:19.380915 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:19.380842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"]
Jul 28 17:25:19.383704 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:25:19.383669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode209cfbb_7730_41c8_af11_d67a21eb9f63.slice/crio-36b3f65de9ddc03c1313a97cc36fa78bad204a90c9d238521520cc43e8df69ed WatchSource:0}: Error finding container 36b3f65de9ddc03c1313a97cc36fa78bad204a90c9d238521520cc43e8df69ed: Status 404 returned error can't find the container with id 36b3f65de9ddc03c1313a97cc36fa78bad204a90c9d238521520cc43e8df69ed
Jul 28 17:25:19.385626 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:19.385611    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:25:19.781150 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:19.781120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" event={"ID":"e209cfbb-7730-41c8-af11-d67a21eb9f63","Type":"ContainerStarted","Data":"36b3f65de9ddc03c1313a97cc36fa78bad204a90c9d238521520cc43e8df69ed"}
Jul 28 17:25:20.632949 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:20.632866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:20.694771 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:20.694714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:21.791388 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:21.791354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" event={"ID":"e209cfbb-7730-41c8-af11-d67a21eb9f63","Type":"ContainerStarted","Data":"e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221"}
Jul 28 17:25:21.791780 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:21.791450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:21.810723 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:21.810681    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podStartSLOduration=2.042823951 podStartE2EDuration="3.810668469s" totalImagesPullingTime="1.767844518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:21.808712644 +0000 UTC m=+876.974348487" watchObservedRunningTime="2026-07-28 17:25:21.810668469 +0000 UTC m=+876.976304302"
Jul 28 17:25:27.802010 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:27.801982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:28.383792 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.383756    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"]
Jul 28 17:25:28.383973 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.383910    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d" containerID="cri-o://e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221" gracePeriod=30
Jul 28 17:25:28.416661 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.416631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:28.531517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.531482    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"]
Jul 28 17:25:28.531743 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.531723    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" containerID="cri-o://8330602f7a79ed17da8cb2523a95c6d7ff44a03875f9a1c0f9699fe0e726cf71" gracePeriod=30
Jul 28 17:25:28.531799 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.531769    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kube-rbac-proxy" containerID="cri-o://58ba6bfa0621fc0ddc83f7f5a709176029475631ed873e30749ed66e445ab394" gracePeriod=30
Jul 28 17:25:28.546339 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.546293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:28.648356 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.648325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:28.676629 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.676601    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"]
Jul 28 17:25:28.679875 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.679861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.682838 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.682809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-80a53-predictor-serving-cert\""
Jul 28 17:25:28.682970 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.682937    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-80a53-kube-rbac-proxy-sar-config\""
Jul 28 17:25:28.691514 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.691493    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"]
Jul 28 17:25:28.691698 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.691681    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container" containerID="cri-o://4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b" gracePeriod=30
Jul 28 17:25:28.691802 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.691778    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kube-rbac-proxy" containerID="cri-o://d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf" gracePeriod=30
Jul 28 17:25:28.695962 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.695934    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"]
Jul 28 17:25:28.704943 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.704917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.705047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.704951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.705047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.704980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.705138 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.705056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4lf\" (UniqueName: \"kubernetes.io/projected/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-api-access-bx4lf\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.705188 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.705160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.740284 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.740260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:28.765406 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.765387    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"]
Jul 28 17:25:28.768752 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.768737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.771545 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.771527    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-80a53-predictor-serving-cert\""
Jul 28 17:25:28.771621 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.771563    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-80a53-kube-rbac-proxy-sar-config\""
Jul 28 17:25:28.778622 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.778603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"]
Jul 28 17:25:28.804692 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.804670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:28.805740 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.805721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.805842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.805753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.805842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.805774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.805842 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.805793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.805974 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:28.805884    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-80a53-predictor-serving-cert" not found
Jul 28 17:25:28.805974 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.805902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.805974 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:28.805936    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls podName:fb9fcab8-6a88-4b8c-9d41-a5e35015ab96 nodeName:}" failed. No retries permitted until 2026-07-28 17:25:29.305918987 +0000 UTC m=+884.471554798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" (UID: "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96") : secret "isvc-sklearn-graph-raw-hpa-80a53-predictor-serving-cert" not found
Jul 28 17:25:28.806095 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.805987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4lf\" (UniqueName: \"kubernetes.io/projected/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-api-access-bx4lf\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.806095 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.806056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.806173 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.806102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp6mj\" (UniqueName: \"kubernetes.io/projected/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-api-access-mp6mj\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.806173 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.806144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9665a9-176e-44d9-a7e7-76989d8bcb62-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.806251 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.806196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.806291 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.806275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.806427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.806410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.806534 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.806515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.814654 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.814633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4lf\" (UniqueName: \"kubernetes.io/projected/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-api-access-bx4lf\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:28.821646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.821615    2578 generic.go:356] "Generic (PLEG): container finished" podID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerID="58ba6bfa0621fc0ddc83f7f5a709176029475631ed873e30749ed66e445ab394" exitCode=2
Jul 28 17:25:28.821740 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.821655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerDied","Data":"58ba6bfa0621fc0ddc83f7f5a709176029475631ed873e30749ed66e445ab394"}
Jul 28 17:25:28.823786 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.823768    2578 generic.go:356] "Generic (PLEG): container finished" podID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerID="d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf" exitCode=2
Jul 28 17:25:28.823876 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.823828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerDied","Data":"d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf"}
Jul 28 17:25:28.865433 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.865404    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:28.907584 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.907527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.907584 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.907563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp6mj\" (UniqueName: \"kubernetes.io/projected/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-api-access-mp6mj\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.907584 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.907582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9665a9-176e-44d9-a7e7-76989d8bcb62-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.907796 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.907606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.907796 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.907636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.907993 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.907971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.908435 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.908413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.908517 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.908412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.910087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.910058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9665a9-176e-44d9-a7e7-76989d8bcb62-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.916508 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.916481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp6mj\" (UniqueName: \"kubernetes.io/projected/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-api-access-mp6mj\") pod \"isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:28.968089 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:28.968061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:29.078385 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.078360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:29.206353 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.206328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"]
Jul 28 17:25:29.208725 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:25:29.208694    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9665a9_176e_44d9_a7e7_76989d8bcb62.slice/crio-0ffc5cc11c404fd0d0f3f6dfd6b8691e384f8c716a3c3108a1e68aee3c3c34e8 WatchSource:0}: Error finding container 0ffc5cc11c404fd0d0f3f6dfd6b8691e384f8c716a3c3108a1e68aee3c3c34e8: Status 404 returned error can't find the container with id 0ffc5cc11c404fd0d0f3f6dfd6b8691e384f8c716a3c3108a1e68aee3c3c34e8
Jul 28 17:25:29.310349 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.310312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:29.312514 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.312494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:29.591777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.591742    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:29.723832 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.723807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"]
Jul 28 17:25:29.726627 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:25:29.726589    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9fcab8_6a88_4b8c_9d41_a5e35015ab96.slice/crio-9f5e43b1bf779f9929c0c7a4d35334f2124331dffd7670047f587887e8570761 WatchSource:0}: Error finding container 9f5e43b1bf779f9929c0c7a4d35334f2124331dffd7670047f587887e8570761: Status 404 returned error can't find the container with id 9f5e43b1bf779f9929c0c7a4d35334f2124331dffd7670047f587887e8570761
Jul 28 17:25:29.830428 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.830386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerStarted","Data":"9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2"}
Jul 28 17:25:29.830867 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.830460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerStarted","Data":"9f5e43b1bf779f9929c0c7a4d35334f2124331dffd7670047f587887e8570761"}
Jul 28 17:25:29.832111 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.832085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerStarted","Data":"34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1"}
Jul 28 17:25:29.832203 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:29.832133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerStarted","Data":"0ffc5cc11c404fd0d0f3f6dfd6b8691e384f8c716a3c3108a1e68aee3c3c34e8"}
Jul 28 17:25:31.406768 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:31.406719    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 28 17:25:31.413222 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:31.413194    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 17:25:32.800228 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:32.800183    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:33.137877 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.137855    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:25:33.246121 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-cabundle-cert\") pod \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") "
Jul 28 17:25:33.246280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-rbac-proxy-sar-config\") pod \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") "
Jul 28 17:25:33.246280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc41b91-8d03-4da5-8999-e07a01af5cfa-proxy-tls\") pod \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") "
Jul 28 17:25:33.246386 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kserve-provision-location\") pod \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") "
Jul 28 17:25:33.246386 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246359    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t276\" (UniqueName: \"kubernetes.io/projected/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-api-access-5t276\") pod \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\" (UID: \"1dc41b91-8d03-4da5-8999-e07a01af5cfa\") "
Jul 28 17:25:33.246548 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1dc41b91-8d03-4da5-8999-e07a01af5cfa" (UID: "1dc41b91-8d03-4da5-8999-e07a01af5cfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:33.246616 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246528    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1dc41b91-8d03-4da5-8999-e07a01af5cfa" (UID: "1dc41b91-8d03-4da5-8999-e07a01af5cfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:33.246723 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dc41b91-8d03-4da5-8999-e07a01af5cfa" (UID: "1dc41b91-8d03-4da5-8999-e07a01af5cfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:33.246774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246719    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:33.246774 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.246745    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:33.248473 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.248452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-api-access-5t276" (OuterVolumeSpecName: "kube-api-access-5t276") pod "1dc41b91-8d03-4da5-8999-e07a01af5cfa" (UID: "1dc41b91-8d03-4da5-8999-e07a01af5cfa"). InnerVolumeSpecName "kube-api-access-5t276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:33.248473 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.248457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc41b91-8d03-4da5-8999-e07a01af5cfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dc41b91-8d03-4da5-8999-e07a01af5cfa" (UID: "1dc41b91-8d03-4da5-8999-e07a01af5cfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:33.347620 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.347595    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc41b91-8d03-4da5-8999-e07a01af5cfa-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:33.347620 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.347622    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:33.347783 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.347631    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t276\" (UniqueName: \"kubernetes.io/projected/1dc41b91-8d03-4da5-8999-e07a01af5cfa-kube-api-access-5t276\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:33.850536 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.850499    2578 generic.go:356] "Generic (PLEG): container finished" podID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerID="8330602f7a79ed17da8cb2523a95c6d7ff44a03875f9a1c0f9699fe0e726cf71" exitCode=0
Jul 28 17:25:33.850906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.850572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerDied","Data":"8330602f7a79ed17da8cb2523a95c6d7ff44a03875f9a1c0f9699fe0e726cf71"}
Jul 28 17:25:33.852357 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.852335    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerID="34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1" exitCode=0
Jul 28 17:25:33.852478 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.852402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerDied","Data":"34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1"}
Jul 28 17:25:33.854179 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.854123    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerID="9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2" exitCode=0
Jul 28 17:25:33.854258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.854188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerDied","Data":"9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2"}
Jul 28 17:25:33.856401 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.856380    2578 generic.go:356] "Generic (PLEG): container finished" podID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerID="4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b" exitCode=0
Jul 28 17:25:33.856501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.856410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerDied","Data":"4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b"}
Jul 28 17:25:33.856501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.856432    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"
Jul 28 17:25:33.856501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.856451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6" event={"ID":"1dc41b91-8d03-4da5-8999-e07a01af5cfa","Type":"ContainerDied","Data":"4f4f2e878c8b2a6d6f8b37ce0b2e9cc3d5aa934733561bf64bc84dd05a9b2cd7"}
Jul 28 17:25:33.856501 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.856480    2578 scope.go:117] "RemoveContainer" containerID="d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf"
Jul 28 17:25:33.870876 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.870854    2578 scope.go:117] "RemoveContainer" containerID="4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b"
Jul 28 17:25:33.880265 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.880246    2578 scope.go:117] "RemoveContainer" containerID="d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e"
Jul 28 17:25:33.887703 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.887632    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"]
Jul 28 17:25:33.891731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.891710    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-02b1d-predictor-7488777cd9-x5fm6"]
Jul 28 17:25:33.893276 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.893217    2578 scope.go:117] "RemoveContainer" containerID="d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf"
Jul 28 17:25:33.893701 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:33.893679    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf\": container with ID starting with d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf not found: ID does not exist" containerID="d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf"
Jul 28 17:25:33.893818 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.893707    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf"} err="failed to get container status \"d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf\": rpc error: code = NotFound desc = could not find container \"d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf\": container with ID starting with d05c5018cca638fa61e5772ecf3e9d67c71d75959cf289f56f1f10b77c4b35bf not found: ID does not exist"
Jul 28 17:25:33.893818 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.893735    2578 scope.go:117] "RemoveContainer" containerID="4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b"
Jul 28 17:25:33.893997 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:33.893979    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b\": container with ID starting with 4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b not found: ID does not exist" containerID="4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b"
Jul 28 17:25:33.894054 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.894001    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b"} err="failed to get container status \"4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b\": rpc error: code = NotFound desc = could not find container \"4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b\": container with ID starting with 4a77b68b2cda87a7749bbb2f88c2adb839ae8acfa9ad9b677678e852150ac89b not found: ID does not exist"
Jul 28 17:25:33.894054 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.894016    2578 scope.go:117] "RemoveContainer" containerID="d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e"
Jul 28 17:25:33.894350 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:33.894326    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e\": container with ID starting with d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e not found: ID does not exist" containerID="d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e"
Jul 28 17:25:33.894441 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.894362    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e"} err="failed to get container status \"d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e\": rpc error: code = NotFound desc = could not find container \"d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e\": container with ID starting with d70bd72ed075aa2b6baa118622ac8b70410fe604ec19acd1fbc7fce835caa35e not found: ID does not exist"
Jul 28 17:25:33.997032 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:33.997005    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:25:34.052834 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.052802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/097f5690-8e64-41a2-ba73-01fc79aeb386-kserve-provision-location\") pod \"097f5690-8e64-41a2-ba73-01fc79aeb386\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") "
Jul 28 17:25:34.053005 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.052867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/097f5690-8e64-41a2-ba73-01fc79aeb386-proxy-tls\") pod \"097f5690-8e64-41a2-ba73-01fc79aeb386\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") "
Jul 28 17:25:34.053005 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.052922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sbxr\" (UniqueName: \"kubernetes.io/projected/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-api-access-8sbxr\") pod \"097f5690-8e64-41a2-ba73-01fc79aeb386\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") "
Jul 28 17:25:34.053005 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.052980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-rbac-proxy-sar-config\") pod \"097f5690-8e64-41a2-ba73-01fc79aeb386\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") "
Jul 28 17:25:34.053005 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.053003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-cabundle-cert\") pod \"097f5690-8e64-41a2-ba73-01fc79aeb386\" (UID: \"097f5690-8e64-41a2-ba73-01fc79aeb386\") "
Jul 28 17:25:34.053193 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.053098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097f5690-8e64-41a2-ba73-01fc79aeb386-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "097f5690-8e64-41a2-ba73-01fc79aeb386" (UID: "097f5690-8e64-41a2-ba73-01fc79aeb386"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:34.053312 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.053275    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "097f5690-8e64-41a2-ba73-01fc79aeb386" (UID: "097f5690-8e64-41a2-ba73-01fc79aeb386"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:34.053384 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.053370    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:34.053439 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.053385    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/097f5690-8e64-41a2-ba73-01fc79aeb386-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:34.053439 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.053390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "097f5690-8e64-41a2-ba73-01fc79aeb386" (UID: "097f5690-8e64-41a2-ba73-01fc79aeb386"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:34.054858 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.054839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097f5690-8e64-41a2-ba73-01fc79aeb386-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "097f5690-8e64-41a2-ba73-01fc79aeb386" (UID: "097f5690-8e64-41a2-ba73-01fc79aeb386"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:34.054995 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.054981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-api-access-8sbxr" (OuterVolumeSpecName: "kube-api-access-8sbxr") pod "097f5690-8e64-41a2-ba73-01fc79aeb386" (UID: "097f5690-8e64-41a2-ba73-01fc79aeb386"). InnerVolumeSpecName "kube-api-access-8sbxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:34.153902 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.153818    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/097f5690-8e64-41a2-ba73-01fc79aeb386-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:34.153902 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.153846    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sbxr\" (UniqueName: \"kubernetes.io/projected/097f5690-8e64-41a2-ba73-01fc79aeb386-kube-api-access-8sbxr\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:34.153902 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.153856    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/097f5690-8e64-41a2-ba73-01fc79aeb386-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:34.863310 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.863257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerStarted","Data":"17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc"}
Jul 28 17:25:34.863719 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.863355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerStarted","Data":"604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f"}
Jul 28 17:25:34.863719 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.863590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:34.865435 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.865410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerStarted","Data":"2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc"}
Jul 28 17:25:34.865553 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.865471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerStarted","Data":"49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27"}
Jul 28 17:25:34.865684 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.865662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:34.865754 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.865719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:34.867491 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.867464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:25:34.868248 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.868221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l" event={"ID":"097f5690-8e64-41a2-ba73-01fc79aeb386","Type":"ContainerDied","Data":"0e537da6c6854c544d7570f448a84b660345b8e4d7ceb3b21308015f79568e9a"}
Jul 28 17:25:34.868365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.868252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"
Jul 28 17:25:34.868365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.868284    2578 scope.go:117] "RemoveContainer" containerID="58ba6bfa0621fc0ddc83f7f5a709176029475631ed873e30749ed66e445ab394"
Jul 28 17:25:34.876294 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.876276    2578 scope.go:117] "RemoveContainer" containerID="8330602f7a79ed17da8cb2523a95c6d7ff44a03875f9a1c0f9699fe0e726cf71"
Jul 28 17:25:34.883039 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.883022    2578 scope.go:117] "RemoveContainer" containerID="ff40ddf287f0e97fd57b69c0c273e699ed6d3a637dc117ac4564349fe7355ce6"
Jul 28 17:25:34.886330 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.886274    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podStartSLOduration=2.879581805 podStartE2EDuration="6.886262435s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00668063s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:34.884976754 +0000 UTC m=+890.050612588" watchObservedRunningTime="2026-07-28 17:25:34.886262435 +0000 UTC m=+890.051898268"
Jul 28 17:25:34.908914 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.908861    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podStartSLOduration=3.094375993 podStartE2EDuration="6.908845512s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814469519s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:34.907720752 +0000 UTC m=+890.073356590" watchObservedRunningTime="2026-07-28 17:25:34.908845512 +0000 UTC m=+890.074481346"
Jul 28 17:25:34.922910 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.922876    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"]
Jul 28 17:25:34.925995 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:34.925972    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-02b1d-predictor-555d69bd44-gck5l"]
Jul 28 17:25:35.362969 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:35.362938    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" path="/var/lib/kubelet/pods/097f5690-8e64-41a2-ba73-01fc79aeb386/volumes"
Jul 28 17:25:35.363459 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:35.363445    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" path="/var/lib/kubelet/pods/1dc41b91-8d03-4da5-8999-e07a01af5cfa/volumes"
Jul 28 17:25:35.873589 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:35.873557    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:35.873951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:35.873641    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:25:35.874959 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:35.874935    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:25:36.877460 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:36.877416    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:25:37.800421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:37.800375    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:40.877639 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:40.877615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:25:40.878112 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:40.878084    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:25:41.882032 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:41.882006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:25:41.882539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:41.882511    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:25:42.800263 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:42.800222    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:42.800433 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:42.800365    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:43.825117 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:43.825089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:47.800692 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:47.800650    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:50.878664 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:50.878623    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:25:51.882999 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:51.882959    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:25:52.800995 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:52.800954    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:57.800621 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:57.800584    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:58.525006 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.524978    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:58.665491 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.665419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e209cfbb-7730-41c8-af11-d67a21eb9f63-openshift-service-ca-bundle\") pod \"e209cfbb-7730-41c8-af11-d67a21eb9f63\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") "
Jul 28 17:25:58.665630 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.665574    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls\") pod \"e209cfbb-7730-41c8-af11-d67a21eb9f63\" (UID: \"e209cfbb-7730-41c8-af11-d67a21eb9f63\") "
Jul 28 17:25:58.665818 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.665797    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e209cfbb-7730-41c8-af11-d67a21eb9f63-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e209cfbb-7730-41c8-af11-d67a21eb9f63" (UID: "e209cfbb-7730-41c8-af11-d67a21eb9f63"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:58.667530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.667506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e209cfbb-7730-41c8-af11-d67a21eb9f63" (UID: "e209cfbb-7730-41c8-af11-d67a21eb9f63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:58.766154 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.766123    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e209cfbb-7730-41c8-af11-d67a21eb9f63-openshift-service-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:58.766154 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.766150    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e209cfbb-7730-41c8-af11-d67a21eb9f63-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:25:58.960814 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.960734    2578 generic.go:356] "Generic (PLEG): container finished" podID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerID="e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221" exitCode=0
Jul 28 17:25:58.960814 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.960780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"
Jul 28 17:25:58.960814 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.960791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" event={"ID":"e209cfbb-7730-41c8-af11-d67a21eb9f63","Type":"ContainerDied","Data":"e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221"}
Jul 28 17:25:58.961355 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.960855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9" event={"ID":"e209cfbb-7730-41c8-af11-d67a21eb9f63","Type":"ContainerDied","Data":"36b3f65de9ddc03c1313a97cc36fa78bad204a90c9d238521520cc43e8df69ed"}
Jul 28 17:25:58.961355 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.960880    2578 scope.go:117] "RemoveContainer" containerID="e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221"
Jul 28 17:25:58.968972 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.968952    2578 scope.go:117] "RemoveContainer" containerID="e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221"
Jul 28 17:25:58.969227 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:25:58.969207    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221\": container with ID starting with e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221 not found: ID does not exist" containerID="e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221"
Jul 28 17:25:58.969283 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.969235    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221"} err="failed to get container status \"e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221\": rpc error: code = NotFound desc = could not find container \"e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221\": container with ID starting with e3fb69bd6e9e3a88cbfb0cf1c76bd640296362cf91960684d7c5d36d71411221 not found: ID does not exist"
Jul 28 17:25:58.983110 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.983077    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"]
Jul 28 17:25:58.987464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:58.987438    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-02b1d-68ddfd4db5-tp8p9"]
Jul 28 17:25:59.362416 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:25:59.362385    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" path="/var/lib/kubelet/pods/e209cfbb-7730-41c8-af11-d67a21eb9f63/volumes"
Jul 28 17:26:00.878866 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:00.878830    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:26:01.882604 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:01.882571    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:26:10.878910 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:10.878869    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:26:11.883175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:11.883134    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:26:20.878748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:20.878711    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:26:21.883405 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:21.883355    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:26:30.878482 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:30.878438    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:26:31.883462 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:31.883433    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:26:40.878444 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:40.878408    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:26:50.878463 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:26:50.878426    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:27:08.656097 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656022    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"]
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656372    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656389    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="storage-initializer"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656395    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="storage-initializer"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656402    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="storage-initializer"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656407    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="storage-initializer"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656413    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656418    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kube-rbac-proxy"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656431    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kube-rbac-proxy"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656440    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656446    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kube-rbac-proxy"
Jul 28 17:27:08.656470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656458    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kube-rbac-proxy"
Jul 28 17:27:08.657057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656503    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kube-rbac-proxy"
Jul 28 17:27:08.657057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656511    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kube-rbac-proxy"
Jul 28 17:27:08.657057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656518    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e209cfbb-7730-41c8-af11-d67a21eb9f63" containerName="model-chainer-raw-02b1d"
Jul 28 17:27:08.657057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656523    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="097f5690-8e64-41a2-ba73-01fc79aeb386" containerName="kserve-container"
Jul 28 17:27:08.657057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.656531    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc41b91-8d03-4da5-8999-e07a01af5cfa" containerName="kserve-container"
Jul 28 17:27:08.659602 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.659584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:08.662269 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.662249    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-80a53-serving-cert\""
Jul 28 17:27:08.662441 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.662270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-80a53-kube-rbac-proxy-sar-config\""
Jul 28 17:27:08.662998 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.662976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:08.668464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.668440    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"]
Jul 28 17:27:08.808966 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.808938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-80a53-bd7b6699c-5z97n\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:08.809118 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.808988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-proxy-tls\") pod \"model-chainer-raw-hpa-80a53-bd7b6699c-5z97n\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:08.909441 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.909362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-80a53-bd7b6699c-5z97n\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:08.909567 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.909438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-proxy-tls\") pod \"model-chainer-raw-hpa-80a53-bd7b6699c-5z97n\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:08.910067 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.910043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-80a53-bd7b6699c-5z97n\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:08.911862 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.911842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-proxy-tls\") pod \"model-chainer-raw-hpa-80a53-bd7b6699c-5z97n\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:08.970798 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:08.970768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:09.113421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:09.113398    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"]
Jul 28 17:27:09.115590 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:27:09.115562    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f59d6c3_e417_4bd3_ac2c_0220fda9e9d3.slice/crio-55f4ef2e93657f52b0473ae608632851b8cd321c582c6024067f92bc0f914119 WatchSource:0}: Error finding container 55f4ef2e93657f52b0473ae608632851b8cd321c582c6024067f92bc0f914119: Status 404 returned error can't find the container with id 55f4ef2e93657f52b0473ae608632851b8cd321c582c6024067f92bc0f914119
Jul 28 17:27:09.237525 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:09.237494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" event={"ID":"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3","Type":"ContainerStarted","Data":"12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778"}
Jul 28 17:27:09.237716 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:09.237542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" event={"ID":"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3","Type":"ContainerStarted","Data":"55f4ef2e93657f52b0473ae608632851b8cd321c582c6024067f92bc0f914119"}
Jul 28 17:27:09.237716 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:09.237585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:09.257209 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:09.257170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podStartSLOduration=1.257157512 podStartE2EDuration="1.257157512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:09.255515372 +0000 UTC m=+984.421151206" watchObservedRunningTime="2026-07-28 17:27:09.257157512 +0000 UTC m=+984.422793345"
Jul 28 17:27:09.994680 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:09.994649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:15.247834 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:15.247806    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:18.707556 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.707521    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"]
Jul 28 17:27:18.708038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.707712    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53" containerID="cri-o://12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778" gracePeriod=30
Jul 28 17:27:18.733622 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.733594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:18.948772 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.948739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:18.988333 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.988120    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"]
Jul 28 17:27:18.991700 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.991678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:18.994716 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.994698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9e866-predictor-serving-cert\""
Jul 28 17:27:18.994834 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:18.994715    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9e866-kube-rbac-proxy-sar-config\""
Jul 28 17:27:19.001666 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.001643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"]
Jul 28 17:27:19.009035 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.009014    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"]
Jul 28 17:27:19.009241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.009207    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" containerID="cri-o://49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27" gracePeriod=30
Jul 28 17:27:19.009371 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.009350    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kube-rbac-proxy" containerID="cri-o://2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc" gracePeriod=30
Jul 28 17:27:19.046838 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.046812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:19.053195 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.053172    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"]
Jul 28 17:27:19.053450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.053425    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" containerID="cri-o://604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f" gracePeriod=30
Jul 28 17:27:19.053551 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.053451    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kube-rbac-proxy" containerID="cri-o://17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc" gracePeriod=30
Jul 28 17:27:19.146335 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.146290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:19.186520 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.186495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.186611 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.186531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.186611 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.186593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-api-access-ssnsq\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.277740 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.277670    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerID="2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc" exitCode=2
Jul 28 17:27:19.277871 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.277743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerDied","Data":"2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc"}
Jul 28 17:27:19.279654 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.279633    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerID="17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc" exitCode=2
Jul 28 17:27:19.279755 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.279708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerDied","Data":"17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc"}
Jul 28 17:27:19.287962 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.287945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-api-access-ssnsq\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.288032 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.288020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.288091 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.288044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.288146 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:19.288133    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-serving-cert: secret "message-dumper-raw-9e866-predictor-serving-cert" not found
Jul 28 17:27:19.288195 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:19.288183    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls podName:00c50c5e-8655-4759-b554-5cb26f7afd9a nodeName:}" failed. No retries permitted until 2026-07-28 17:27:19.788169017 +0000 UTC m=+994.953804830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls") pod "message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" (UID: "00c50c5e-8655-4759-b554-5cb26f7afd9a") : secret "message-dumper-raw-9e866-predictor-serving-cert" not found
Jul 28 17:27:19.288647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.288626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.297779 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.297759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-api-access-ssnsq\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.347856 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.347830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:19.792454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.792408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.794946 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.794909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls\") pod \"message-dumper-raw-9e866-predictor-7c59699df8-8gxhp\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:19.904572 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:19.904532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:20.029317 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:20.029279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"]
Jul 28 17:27:20.031812 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:27:20.031783    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c50c5e_8655_4759_b554_5cb26f7afd9a.slice/crio-caae42b97b18ac65e14ccff2ab33b7f37f75a882c33da6515f487b0bbf4bdd1c WatchSource:0}: Error finding container caae42b97b18ac65e14ccff2ab33b7f37f75a882c33da6515f487b0bbf4bdd1c: Status 404 returned error can't find the container with id caae42b97b18ac65e14ccff2ab33b7f37f75a882c33da6515f487b0bbf4bdd1c
Jul 28 17:27:20.246552 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:20.246509    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:20.289368 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:20.289333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" event={"ID":"00c50c5e-8655-4759-b554-5cb26f7afd9a","Type":"ContainerStarted","Data":"caae42b97b18ac65e14ccff2ab33b7f37f75a882c33da6515f487b0bbf4bdd1c"}
Jul 28 17:27:20.874070 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:20.874025    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 17:27:20.878417 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:20.878389    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:27:21.878025 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:21.877984    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 17:27:21.883344 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:21.883319    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:27:22.299552 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.299446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" event={"ID":"00c50c5e-8655-4759-b554-5cb26f7afd9a","Type":"ContainerStarted","Data":"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"}
Jul 28 17:27:22.299552 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.299524    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:22.299552 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.299552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" event={"ID":"00c50c5e-8655-4759-b554-5cb26f7afd9a","Type":"ContainerStarted","Data":"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"}
Jul 28 17:27:22.299904 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.299704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:22.301450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.301427    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:22.320405 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.320364    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" podStartSLOduration=3.083102456 podStartE2EDuration="4.320350472s" totalImagesPullingTime="1.237248016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:22.318744922 +0000 UTC m=+997.484380760" watchObservedRunningTime="2026-07-28 17:27:22.320350472 +0000 UTC m=+997.485986305"
Jul 28 17:27:22.698859 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.698836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:27:22.717906 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.717886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp6mj\" (UniqueName: \"kubernetes.io/projected/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-api-access-mp6mj\") pod \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") "
Jul 28 17:27:22.718012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.717927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-rbac-proxy-sar-config\") pod \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") "
Jul 28 17:27:22.718012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.717958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-cabundle-cert\") pod \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") "
Jul 28 17:27:22.718012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.717986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kserve-provision-location\") pod \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") "
Jul 28 17:27:22.718012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.718005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9665a9-176e-44d9-a7e7-76989d8bcb62-proxy-tls\") pod \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\" (UID: \"7b9665a9-176e-44d9-a7e7-76989d8bcb62\") "
Jul 28 17:27:22.718341 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.718285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b9665a9-176e-44d9-a7e7-76989d8bcb62" (UID: "7b9665a9-176e-44d9-a7e7-76989d8bcb62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:27:22.718444 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.718353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b9665a9-176e-44d9-a7e7-76989d8bcb62" (UID: "7b9665a9-176e-44d9-a7e7-76989d8bcb62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:27:22.718444 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.718359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b9665a9-176e-44d9-a7e7-76989d8bcb62" (UID: "7b9665a9-176e-44d9-a7e7-76989d8bcb62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:27:22.719839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.719819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-api-access-mp6mj" (OuterVolumeSpecName: "kube-api-access-mp6mj") pod "7b9665a9-176e-44d9-a7e7-76989d8bcb62" (UID: "7b9665a9-176e-44d9-a7e7-76989d8bcb62"). InnerVolumeSpecName "kube-api-access-mp6mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:27:22.719930 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.719893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9665a9-176e-44d9-a7e7-76989d8bcb62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b9665a9-176e-44d9-a7e7-76989d8bcb62" (UID: "7b9665a9-176e-44d9-a7e7-76989d8bcb62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:27:22.818685 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.818660    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp6mj\" (UniqueName: \"kubernetes.io/projected/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-api-access-mp6mj\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:22.818685 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.818684    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:22.818872 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.818694    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9665a9-176e-44d9-a7e7-76989d8bcb62-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:22.818872 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.818704    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9665a9-176e-44d9-a7e7-76989d8bcb62-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:22.818872 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:22.818713    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9665a9-176e-44d9-a7e7-76989d8bcb62-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:23.305718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.305691    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerID="604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f" exitCode=0
Jul 28 17:27:23.306159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.305745    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"
Jul 28 17:27:23.306159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.305735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerDied","Data":"604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f"}
Jul 28 17:27:23.306159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.305874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq" event={"ID":"7b9665a9-176e-44d9-a7e7-76989d8bcb62","Type":"ContainerDied","Data":"0ffc5cc11c404fd0d0f3f6dfd6b8691e384f8c716a3c3108a1e68aee3c3c34e8"}
Jul 28 17:27:23.306159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.305913    2578 scope.go:117] "RemoveContainer" containerID="17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc"
Jul 28 17:27:23.314346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.314289    2578 scope.go:117] "RemoveContainer" containerID="604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f"
Jul 28 17:27:23.321493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.321477    2578 scope.go:117] "RemoveContainer" containerID="34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1"
Jul 28 17:27:23.329132 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.329105    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"]
Jul 28 17:27:23.329376 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.329363    2578 scope.go:117] "RemoveContainer" containerID="17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc"
Jul 28 17:27:23.329626 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:23.329607    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc\": container with ID starting with 17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc not found: ID does not exist" containerID="17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc"
Jul 28 17:27:23.329670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.329636    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc"} err="failed to get container status \"17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc\": rpc error: code = NotFound desc = could not find container \"17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc\": container with ID starting with 17d5ab3a7f747b141a368dcc6f6cb7ad08119605deeb27a1dee98484a4cd56dc not found: ID does not exist"
Jul 28 17:27:23.329670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.329653    2578 scope.go:117] "RemoveContainer" containerID="604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f"
Jul 28 17:27:23.329899 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:23.329886    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f\": container with ID starting with 604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f not found: ID does not exist" containerID="604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f"
Jul 28 17:27:23.329952 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.329902    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f"} err="failed to get container status \"604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f\": rpc error: code = NotFound desc = could not find container \"604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f\": container with ID starting with 604b2dab3bf96cc360e49aad35ab709d62f4c3a01600bd0a432f6650fffd507f not found: ID does not exist"
Jul 28 17:27:23.329952 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.329918    2578 scope.go:117] "RemoveContainer" containerID="34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1"
Jul 28 17:27:23.330129 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:23.330113    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1\": container with ID starting with 34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1 not found: ID does not exist" containerID="34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1"
Jul 28 17:27:23.330165 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.330136    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1"} err="failed to get container status \"34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1\": rpc error: code = NotFound desc = could not find container \"34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1\": container with ID starting with 34d1069b4c0d0683e5f01855eb74a63672c7eecfde4b848e7821078f44b84ba1 not found: ID does not exist"
Jul 28 17:27:23.336217 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.336195    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-80a53-predictor-857c6b897-pfqtq"]
Jul 28 17:27:23.362651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.362629    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" path="/var/lib/kubelet/pods/7b9665a9-176e-44d9-a7e7-76989d8bcb62/volumes"
Jul 28 17:27:23.650704 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.650677    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:27:23.725893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.725865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls\") pod \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") "
Jul 28 17:27:23.726057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.725911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-cabundle-cert\") pod \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") "
Jul 28 17:27:23.726057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.725942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kserve-provision-location\") pod \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") "
Jul 28 17:27:23.726057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.725983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-rbac-proxy-sar-config\") pod \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") "
Jul 28 17:27:23.726222 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.726064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx4lf\" (UniqueName: \"kubernetes.io/projected/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-api-access-bx4lf\") pod \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\" (UID: \"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96\") "
Jul 28 17:27:23.726333 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.726289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" (UID: "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:27:23.726411 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.726364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" (UID: "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:27:23.726411 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.726383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" (UID: "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:27:23.727932 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.727904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" (UID: "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:27:23.728039 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.727977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-api-access-bx4lf" (OuterVolumeSpecName: "kube-api-access-bx4lf") pod "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" (UID: "fb9fcab8-6a88-4b8c-9d41-a5e35015ab96"). InnerVolumeSpecName "kube-api-access-bx4lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:27:23.827544 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.827482    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4lf\" (UniqueName: \"kubernetes.io/projected/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-api-access-bx4lf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:23.827544 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.827505    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:23.827544 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.827515    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:23.827544 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.827522    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:23.827544 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:23.827530    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:24.311908 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.311828    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerID="49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27" exitCode=0
Jul 28 17:27:24.311908 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.311881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"
Jul 28 17:27:24.312437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.311919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerDied","Data":"49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27"}
Jul 28 17:27:24.312437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.311994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg" event={"ID":"fb9fcab8-6a88-4b8c-9d41-a5e35015ab96","Type":"ContainerDied","Data":"9f5e43b1bf779f9929c0c7a4d35334f2124331dffd7670047f587887e8570761"}
Jul 28 17:27:24.312437 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.312039    2578 scope.go:117] "RemoveContainer" containerID="2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc"
Jul 28 17:27:24.321967 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.321949    2578 scope.go:117] "RemoveContainer" containerID="49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27"
Jul 28 17:27:24.329618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.329602    2578 scope.go:117] "RemoveContainer" containerID="9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2"
Jul 28 17:27:24.336513 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.336496    2578 scope.go:117] "RemoveContainer" containerID="2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc"
Jul 28 17:27:24.336801 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:24.336779    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc\": container with ID starting with 2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc not found: ID does not exist" containerID="2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc"
Jul 28 17:27:24.336869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.336812    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc"} err="failed to get container status \"2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc\": rpc error: code = NotFound desc = could not find container \"2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc\": container with ID starting with 2fa6964974b3a024927d9cd5b6f9ce1c09a809bc7e11700fff4b137f36113ecc not found: ID does not exist"
Jul 28 17:27:24.336869 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.336839    2578 scope.go:117] "RemoveContainer" containerID="49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27"
Jul 28 17:27:24.337104 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:24.337086    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27\": container with ID starting with 49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27 not found: ID does not exist" containerID="49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27"
Jul 28 17:27:24.337162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.337107    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27"} err="failed to get container status \"49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27\": rpc error: code = NotFound desc = could not find container \"49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27\": container with ID starting with 49de100f24fe43aeb3ca87310559e6a2d7dd3ef9588aff87f1357240f8df2d27 not found: ID does not exist"
Jul 28 17:27:24.337162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.337122    2578 scope.go:117] "RemoveContainer" containerID="9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2"
Jul 28 17:27:24.337371 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:24.337353    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2\": container with ID starting with 9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2 not found: ID does not exist" containerID="9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2"
Jul 28 17:27:24.337427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.337376    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2"} err="failed to get container status \"9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2\": rpc error: code = NotFound desc = could not find container \"9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2\": container with ID starting with 9a5d2a4e91f9cfa4171354f1ee2a00e720ce3d51aaeec4d4dd35021f06cdf9f2 not found: ID does not exist"
Jul 28 17:27:24.337463 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.337439    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"]
Jul 28 17:27:24.340957 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:24.340935    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-80a53-predictor-6f49f85b9b-2fkzg"]
Jul 28 17:27:25.245813 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:25.245778    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:25.362955 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:25.362926    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" path="/var/lib/kubelet/pods/fb9fcab8-6a88-4b8c-9d41-a5e35015ab96/volumes"
Jul 28 17:27:29.316567 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:29.316541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:27:30.246603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:30.246567    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:30.246771 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:30.246676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:35.246417 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:35.246381    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:38.918754 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.918715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:38.956237 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956211    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"]
Jul 28 17:27:38.956513 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956500    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container"
Jul 28 17:27:38.956566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956514    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container"
Jul 28 17:27:38.956566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956530    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="storage-initializer"
Jul 28 17:27:38.956566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956535    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="storage-initializer"
Jul 28 17:27:38.956566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956544    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kube-rbac-proxy"
Jul 28 17:27:38.956566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956549    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kube-rbac-proxy"
Jul 28 17:27:38.956566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956561    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="storage-initializer"
Jul 28 17:27:38.956566 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="storage-initializer"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956572    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956597    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956604    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kube-rbac-proxy"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956609    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kube-rbac-proxy"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956654    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kube-rbac-proxy"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956665    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9665a9-176e-44d9-a7e7-76989d8bcb62" containerName="kserve-container"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956673    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kube-rbac-proxy"
Jul 28 17:27:38.956766 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.956680    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9fcab8-6a88-4b8c-9d41-a5e35015ab96" containerName="kserve-container"
Jul 28 17:27:38.964231 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.964211    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:38.967203 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.967180    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:27:38.967321 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.967180    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9e866-kube-rbac-proxy-sar-config\""
Jul 28 17:27:38.967321 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.967238    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:27:38.967321 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.967238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9e866-predictor-serving-cert\""
Jul 28 17:27:38.972714 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:38.972692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"]
Jul 28 17:27:39.041821 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.041788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-proxy-tls\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.041952 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.041828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kserve-provision-location\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.041952 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.041855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-api-access-pgndx\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.041952 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.041934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-cabundle-cert\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.042139 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.042025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.136099 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.136069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:39.143047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.143162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-proxy-tls\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.143162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kserve-provision-location\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.143162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-api-access-pgndx\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.143341 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-cabundle-cert\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.143523 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kserve-provision-location\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.143765 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.143843 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.143750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-cabundle-cert\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.145484 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.145467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-proxy-tls\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.151838 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.151814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-api-access-pgndx\") pod \"isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.274369 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.274295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:39.405246 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:39.405093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"]
Jul 28 17:27:39.407415 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:27:39.407391    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c06af1_1cfc_4dd0_8e3a_b1c35caa7336.slice/crio-5e15684b0256f863417c6141e1ac45627f71a4200a8a88c5bf83038787c3da93 WatchSource:0}: Error finding container 5e15684b0256f863417c6141e1ac45627f71a4200a8a88c5bf83038787c3da93: Status 404 returned error can't find the container with id 5e15684b0256f863417c6141e1ac45627f71a4200a8a88c5bf83038787c3da93
Jul 28 17:27:40.246843 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:40.246794    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:40.376870 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:40.376836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerStarted","Data":"f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a"}
Jul 28 17:27:40.377006 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:40.376900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerStarted","Data":"5e15684b0256f863417c6141e1ac45627f71a4200a8a88c5bf83038787c3da93"}
Jul 28 17:27:43.389125 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:43.389039    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerID="f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a" exitCode=0
Jul 28 17:27:43.389125 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:43.389111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerDied","Data":"f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a"}
Jul 28 17:27:44.400832 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.400795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerStarted","Data":"2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742"}
Jul 28 17:27:44.401209 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.400885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerStarted","Data":"b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8"}
Jul 28 17:27:44.401209 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.400911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerStarted","Data":"e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153"}
Jul 28 17:27:44.401209 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.401178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:44.401373 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.401317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:44.401373 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.401350    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:44.402761 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.402732    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:27:44.403483 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.403444    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:44.425132 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:44.425093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podStartSLOduration=2.785716602 podStartE2EDuration="6.425081434s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.639364832s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:44.423714034 +0000 UTC m=+1019.589349867" watchObservedRunningTime="2026-07-28 17:27:44.425081434 +0000 UTC m=+1019.590717268"
Jul 28 17:27:45.246106 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:45.246070    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:45.405128 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:45.405086    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:27:45.405636 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:45.405613    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:48.885045 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:48.885017    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:49.019990 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.019910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-proxy-tls\") pod \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") "
Jul 28 17:27:49.020155 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.019997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-openshift-service-ca-bundle\") pod \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\" (UID: \"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3\") "
Jul 28 17:27:49.020370 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.020342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" (UID: "1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:27:49.021992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.021972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" (UID: "1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:27:49.120686 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.120663    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-openshift-service-ca-bundle\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.120686 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.120684    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.421201 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.421173    2578 generic.go:356] "Generic (PLEG): container finished" podID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerID="12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778" exitCode=137
Jul 28 17:27:49.421350 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.421222    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"
Jul 28 17:27:49.421350 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.421225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" event={"ID":"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3","Type":"ContainerDied","Data":"12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778"}
Jul 28 17:27:49.421350 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.421273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n" event={"ID":"1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3","Type":"ContainerDied","Data":"55f4ef2e93657f52b0473ae608632851b8cd321c582c6024067f92bc0f914119"}
Jul 28 17:27:49.421350 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.421294    2578 scope.go:117] "RemoveContainer" containerID="12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778"
Jul 28 17:27:49.428992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.428977    2578 scope.go:117] "RemoveContainer" containerID="12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778"
Jul 28 17:27:49.429253 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:27:49.429235    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778\": container with ID starting with 12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778 not found: ID does not exist" containerID="12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778"
Jul 28 17:27:49.429320 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.429261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778"} err="failed to get container status \"12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778\": rpc error: code = NotFound desc = could not find container \"12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778\": container with ID starting with 12d3e665e298c67ba1a4c88ff3e91496bc6f0a55e6f7e9d384137e07d96fd778 not found: ID does not exist"
Jul 28 17:27:49.438500 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.438477    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"]
Jul 28 17:27:49.446022 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:49.443409    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-80a53-bd7b6699c-5z97n"]
Jul 28 17:27:50.409893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:50.409856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:27:50.410431 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:50.410397    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:27:50.410725 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:50.410701    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:51.363279 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:27:51.363243    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" path="/var/lib/kubelet/pods/1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3/volumes"
Jul 28 17:28:00.170676 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:00.170640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:00.411324 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:00.411270    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:28:00.411718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:00.411689    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:10.410972 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:10.410931    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:28:10.411445 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:10.411367    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:20.411112 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:20.411072    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:28:20.411601 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:20.411575    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:30.411224 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:30.411180    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:28:30.411710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:30.411613    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:40.411131 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:40.411028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:28:40.411617 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:40.411569    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:50.410715 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:50.410677    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:28:50.411163 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:28:50.411123    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:00.411488 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:00.411444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:29:00.411927 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:00.411778    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:29:13.958506 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:13.958479    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-9e866-predictor-7c59699df8-8gxhp_00c50c5e-8655-4759-b554-5cb26f7afd9a/kserve-container/0.log"
Jul 28 17:29:14.255354 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.255261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:14.290498 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.290469    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"]
Jul 28 17:29:14.290773 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.290736    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" containerID="cri-o://2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742" gracePeriod=30
Jul 28 17:29:14.290773 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.290763    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy" containerID="cri-o://b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8" gracePeriod=30
Jul 28 17:29:14.290975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.290732    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" containerID="cri-o://e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153" gracePeriod=30
Jul 28 17:29:14.326385 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.326358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"]
Jul 28 17:29:14.326734 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.326717    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53"
Jul 28 17:29:14.326817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.326737    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53"
Jul 28 17:29:14.326875 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.326825    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f59d6c3-e417-4bd3-ac2c-0220fda9e9d3" containerName="model-chainer-raw-hpa-80a53"
Jul 28 17:29:14.329964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.329947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.331553 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.331498    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:14.334062 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.334045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-ffe87-predictor-serving-cert\""
Jul 28 17:29:14.334712 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.334692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-ffe87-kube-rbac-proxy-sar-config\""
Jul 28 17:29:14.342948 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.342927    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"]
Jul 28 17:29:14.349436 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.349416    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"]
Jul 28 17:29:14.349651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.349630    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kserve-container" containerID="cri-o://35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc" gracePeriod=30
Jul 28 17:29:14.349747 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.349726    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kube-rbac-proxy" containerID="cri-o://52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0" gracePeriod=30
Jul 28 17:29:14.486964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.486932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-api-access-h2d47\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.487086 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.486973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e2d7d-7ded-49f8-b634-163710f392d7-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.487137 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.487094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-cabundle-cert\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.487178 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.487152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.487212 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.487195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.587784 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.587751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-cabundle-cert\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.587784 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.587785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.587953 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.587811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.587953 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.587839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-api-access-h2d47\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.587953 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.587857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e2d7d-7ded-49f8-b634-163710f392d7-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.587953 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:14.587922    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-serving-cert: secret "isvc-sklearn-scale-raw-ffe87-predictor-serving-cert" not found
Jul 28 17:29:14.588146 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:14.587986    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls podName:1c0e2d7d-7ded-49f8-b634-163710f392d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:29:15.087965744 +0000 UTC m=+1110.253601579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls") pod "isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" (UID: "1c0e2d7d-7ded-49f8-b634-163710f392d7") : secret "isvc-sklearn-scale-raw-ffe87-predictor-serving-cert" not found
Jul 28 17:29:14.588337 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.588318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e2d7d-7ded-49f8-b634-163710f392d7-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.588568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.588547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-cabundle-cert\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.588706 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.588565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.597912 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.597889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-api-access-h2d47\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:14.600897 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.600876    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:29:14.626498 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.626461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:14.688688 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.688665    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls\") pod \"00c50c5e-8655-4759-b554-5cb26f7afd9a\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") "
Jul 28 17:29:14.688816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.688754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-rbac-proxy-sar-config\") pod \"00c50c5e-8655-4759-b554-5cb26f7afd9a\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") "
Jul 28 17:29:14.688816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.688782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-api-access-ssnsq\") pod \"00c50c5e-8655-4759-b554-5cb26f7afd9a\" (UID: \"00c50c5e-8655-4759-b554-5cb26f7afd9a\") "
Jul 28 17:29:14.689089 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.689063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00c50c5e-8655-4759-b554-5cb26f7afd9a" (UID: "00c50c5e-8655-4759-b554-5cb26f7afd9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:29:14.690708 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.690686    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-api-access-ssnsq" (OuterVolumeSpecName: "kube-api-access-ssnsq") pod "00c50c5e-8655-4759-b554-5cb26f7afd9a" (UID: "00c50c5e-8655-4759-b554-5cb26f7afd9a"). InnerVolumeSpecName "kube-api-access-ssnsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:29:14.690793 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.690751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00c50c5e-8655-4759-b554-5cb26f7afd9a" (UID: "00c50c5e-8655-4759-b554-5cb26f7afd9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:29:14.746206 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.746180    2578 generic.go:356] "Generic (PLEG): container finished" podID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerID="52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0" exitCode=2
Jul 28 17:29:14.746206 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.746202    2578 generic.go:356] "Generic (PLEG): container finished" podID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerID="35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc" exitCode=2
Jul 28 17:29:14.746395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.746211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" event={"ID":"00c50c5e-8655-4759-b554-5cb26f7afd9a","Type":"ContainerDied","Data":"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"}
Jul 28 17:29:14.746395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.746230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"
Jul 28 17:29:14.746395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.746291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" event={"ID":"00c50c5e-8655-4759-b554-5cb26f7afd9a","Type":"ContainerDied","Data":"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"}
Jul 28 17:29:14.746395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.746344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp" event={"ID":"00c50c5e-8655-4759-b554-5cb26f7afd9a","Type":"ContainerDied","Data":"caae42b97b18ac65e14ccff2ab33b7f37f75a882c33da6515f487b0bbf4bdd1c"}
Jul 28 17:29:14.746395 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.746330    2578 scope.go:117] "RemoveContainer" containerID="52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"
Jul 28 17:29:14.748850 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.748832    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerID="b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8" exitCode=2
Jul 28 17:29:14.748936 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.748911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerDied","Data":"b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8"}
Jul 28 17:29:14.754605 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.754590    2578 scope.go:117] "RemoveContainer" containerID="35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"
Jul 28 17:29:14.761629 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.761614    2578 scope.go:117] "RemoveContainer" containerID="52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"
Jul 28 17:29:14.761875 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:14.761856    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0\": container with ID starting with 52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0 not found: ID does not exist" containerID="52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"
Jul 28 17:29:14.761923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.761883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"} err="failed to get container status \"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0\": rpc error: code = NotFound desc = could not find container \"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0\": container with ID starting with 52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0 not found: ID does not exist"
Jul 28 17:29:14.761923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.761899    2578 scope.go:117] "RemoveContainer" containerID="35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"
Jul 28 17:29:14.762122 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:14.762105    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc\": container with ID starting with 35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc not found: ID does not exist" containerID="35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"
Jul 28 17:29:14.762169 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.762127    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"} err="failed to get container status \"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc\": rpc error: code = NotFound desc = could not find container \"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc\": container with ID starting with 35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc not found: ID does not exist"
Jul 28 17:29:14.762169 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.762142    2578 scope.go:117] "RemoveContainer" containerID="52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"
Jul 28 17:29:14.762365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.762349    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0"} err="failed to get container status \"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0\": rpc error: code = NotFound desc = could not find container \"52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0\": container with ID starting with 52f33fff28141098dccd3a93abdcfaef6b7d4ed91940cb3401870d130a9508e0 not found: ID does not exist"
Jul 28 17:29:14.762415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.762365    2578 scope.go:117] "RemoveContainer" containerID="35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"
Jul 28 17:29:14.762558 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.762543    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc"} err="failed to get container status \"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc\": rpc error: code = NotFound desc = could not find container \"35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc\": container with ID starting with 35e34860c1540244c540fb61e9e40908cd47c2764b5b2850f482e6cc336b59cc not found: ID does not exist"
Jul 28 17:29:14.772212 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.772156    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"]
Jul 28 17:29:14.780219 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.780198    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9e866-predictor-7c59699df8-8gxhp"]
Jul 28 17:29:14.790113 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.790097    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c50c5e-8655-4759-b554-5cb26f7afd9a-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:14.790177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.790116    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:14.790177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.790126    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/00c50c5e-8655-4759-b554-5cb26f7afd9a-kube-api-access-ssnsq\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:14.820841 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:14.820815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:15.091388 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.091355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:15.093702 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.093678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls\") pod \"isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:15.241538 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.241494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:15.362506 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.362439    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" path="/var/lib/kubelet/pods/00c50c5e-8655-4759-b554-5cb26f7afd9a/volumes"
Jul 28 17:29:15.368856 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.368830    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"]
Jul 28 17:29:15.371208 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:29:15.371182    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0e2d7d_7ded_49f8_b634_163710f392d7.slice/crio-49ce532791ef7762465de982ae03b8c080d0d17e8fe88796483ebbcf4f5523b9 WatchSource:0}: Error finding container 49ce532791ef7762465de982ae03b8c080d0d17e8fe88796483ebbcf4f5523b9: Status 404 returned error can't find the container with id 49ce532791ef7762465de982ae03b8c080d0d17e8fe88796483ebbcf4f5523b9
Jul 28 17:29:15.405621 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.405582    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 17:29:15.755189 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.755109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerStarted","Data":"2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55"}
Jul 28 17:29:15.755189 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:15.755175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerStarted","Data":"49ce532791ef7762465de982ae03b8c080d0d17e8fe88796483ebbcf4f5523b9"}
Jul 28 17:29:19.769668 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:19.769631    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerID="2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55" exitCode=0
Jul 28 17:29:19.770186 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:19.769706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerDied","Data":"2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55"}
Jul 28 17:29:19.772242 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:19.772211    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerID="e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153" exitCode=0
Jul 28 17:29:19.772368 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:19.772241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerDied","Data":"e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153"}
Jul 28 17:29:20.406087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:20.406053    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 17:29:20.410404 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:20.410377    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:29:20.410743 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:20.410720    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:20.778026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:20.777943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerStarted","Data":"c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2"}
Jul 28 17:29:20.778393 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:20.778030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerStarted","Data":"541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5"}
Jul 28 17:29:20.778393 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:20.778244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:20.799675 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:20.799628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podStartSLOduration=2.909577336 podStartE2EDuration="6.799614706s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.89003737s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:20.798486729 +0000 UTC m=+1115.964122563" watchObservedRunningTime="2026-07-28 17:29:20.799614706 +0000 UTC m=+1115.965250539"
Jul 28 17:29:21.781980 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:21.781952    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:21.783235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:21.783208    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:29:22.785238 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:22.785202    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:29:25.405559 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:25.405518    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 17:29:25.405918 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:25.405622    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:29:27.789737 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:27.789704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:29:27.790376 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:27.790348    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:29:30.405754 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:30.405710    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 17:29:30.411083 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:30.411049    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:29:30.411383 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:30.411362    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:35.405765 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:35.405724    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 17:29:37.790540 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:37.790501    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:29:40.410406 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:40.410358    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 17:29:40.410872 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:40.410401    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:29:40.410939 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:40.410878    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:29:40.411204 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:40.411184    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:29:40.411276 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:40.411266    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:29:44.433811 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.433780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:29:44.522147 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-rbac-proxy-sar-config\") pod \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") "
Jul 28 17:29:44.522336 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522150    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-api-access-pgndx\") pod \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") "
Jul 28 17:29:44.522336 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kserve-provision-location\") pod \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") "
Jul 28 17:29:44.522336 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-proxy-tls\") pod \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") "
Jul 28 17:29:44.522336 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522281    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-cabundle-cert\") pod \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\" (UID: \"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336\") "
Jul 28 17:29:44.522557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" (UID: "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:29:44.522613 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522587    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" (UID: "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:29:44.522649 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.522615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" (UID: "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:29:44.524324 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.524288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" (UID: "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:29:44.524324 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.524289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-api-access-pgndx" (OuterVolumeSpecName: "kube-api-access-pgndx") pod "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" (UID: "e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336"). InnerVolumeSpecName "kube-api-access-pgndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:29:44.622816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.622789    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:44.622816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.622813    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:44.622987 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.622827    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kube-api-access-pgndx\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:44.622987 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.622840    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:44.622987 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.622851    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:29:44.872801 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.872765    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerID="2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742" exitCode=0
Jul 28 17:29:44.872983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.872836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"
Jul 28 17:29:44.872983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.872850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerDied","Data":"2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742"}
Jul 28 17:29:44.872983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.872916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8" event={"ID":"e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336","Type":"ContainerDied","Data":"5e15684b0256f863417c6141e1ac45627f71a4200a8a88c5bf83038787c3da93"}
Jul 28 17:29:44.872983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.872947    2578 scope.go:117] "RemoveContainer" containerID="2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742"
Jul 28 17:29:44.886447 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.886430    2578 scope.go:117] "RemoveContainer" containerID="b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8"
Jul 28 17:29:44.893838 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.893816    2578 scope.go:117] "RemoveContainer" containerID="e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153"
Jul 28 17:29:44.901213 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.901191    2578 scope.go:117] "RemoveContainer" containerID="f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a"
Jul 28 17:29:44.901762 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.901742    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"]
Jul 28 17:29:44.906446 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.906427    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9e866-predictor-7f8cb745cf-pc2w8"]
Jul 28 17:29:44.908705 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.908687    2578 scope.go:117] "RemoveContainer" containerID="2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742"
Jul 28 17:29:44.908955 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:44.908936    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742\": container with ID starting with 2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742 not found: ID does not exist" containerID="2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742"
Jul 28 17:29:44.909026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.908967    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742"} err="failed to get container status \"2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742\": rpc error: code = NotFound desc = could not find container \"2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742\": container with ID starting with 2cb48d2a31f8b529803113e0ace189b76c016efc573b0e5d39aeb298e38d7742 not found: ID does not exist"
Jul 28 17:29:44.909026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.908991    2578 scope.go:117] "RemoveContainer" containerID="b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8"
Jul 28 17:29:44.909228 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:44.909213    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8\": container with ID starting with b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8 not found: ID does not exist" containerID="b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8"
Jul 28 17:29:44.909269 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.909234    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8"} err="failed to get container status \"b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8\": rpc error: code = NotFound desc = could not find container \"b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8\": container with ID starting with b147f3ec312d0110c75e8bdad21b4e7bad0b632803bb0c771628138281696fc8 not found: ID does not exist"
Jul 28 17:29:44.909269 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.909247    2578 scope.go:117] "RemoveContainer" containerID="e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153"
Jul 28 17:29:44.909487 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:44.909472    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153\": container with ID starting with e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153 not found: ID does not exist" containerID="e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153"
Jul 28 17:29:44.909546 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.909490    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153"} err="failed to get container status \"e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153\": rpc error: code = NotFound desc = could not find container \"e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153\": container with ID starting with e763d9aea5572084c4ce76ff4740674e1a398e8c1ec019da109e8b6558e03153 not found: ID does not exist"
Jul 28 17:29:44.909546 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.909501    2578 scope.go:117] "RemoveContainer" containerID="f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a"
Jul 28 17:29:44.909733 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:29:44.909714    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a\": container with ID starting with f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a not found: ID does not exist" containerID="f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a"
Jul 28 17:29:44.909791 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:44.909741    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a"} err="failed to get container status \"f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a\": rpc error: code = NotFound desc = could not find container \"f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a\": container with ID starting with f8026a5c7e2784db700e3d3c258f734de0bbb5cc38aaaaa7e88dfa521bf6941a not found: ID does not exist"
Jul 28 17:29:45.362979 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:45.362946    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" path="/var/lib/kubelet/pods/e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336/volumes"
Jul 28 17:29:47.790989 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:47.790945    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:29:57.791189 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:29:57.791149    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:30:07.790464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:30:07.790375    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:30:17.791224 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:30:17.791185    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:30:27.790749 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:30:27.790709    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:30:37.790951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:30:37.790906    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:30:38.358658 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:30:38.358617    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:30:48.358754 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:30:48.358715    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:30:58.358639 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:30:58.358601    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:08.358711 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:08.358675    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:18.359093 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:18.359046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:28.358598 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:28.358558    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:38.358575 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:38.358491    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:48.360098 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:48.360067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:31:54.439361 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:54.439329    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"]
Jul 28 17:31:54.439833 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:54.439590    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" containerID="cri-o://541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5" gracePeriod=30
Jul 28 17:31:54.439833 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:54.439628    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kube-rbac-proxy" containerID="cri-o://c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2" gracePeriod=30
Jul 28 17:31:54.502214 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:54.502176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:55.364801 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:55.364768    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerID="c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2" exitCode=2
Jul 28 17:31:55.364983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:55.364840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerDied","Data":"c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2"}
Jul 28 17:31:57.785891 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:57.785853    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 17:31:58.359205 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:58.359157    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:59.551338 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.551284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:59.606012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.605981    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"]
Jul 28 17:31:59.606269 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606257    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kserve-container"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606271    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kserve-container"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606281    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kube-rbac-proxy"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606287    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kube-rbac-proxy"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606295    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606314    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606327    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606334    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container"
Jul 28 17:31:59.606346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="storage-initializer"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606354    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="storage-initializer"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606362    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606366    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606411    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kube-rbac-proxy"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606421    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="agent"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606426    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kube-rbac-proxy"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606433    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c50c5e-8655-4759-b554-5cb26f7afd9a" containerName="kserve-container"
Jul 28 17:31:59.606586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.606438    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c06af1-1cfc-4dd0-8e3a-b1c35caa7336" containerName="kserve-container"
Jul 28 17:31:59.609464 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.609444    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"]
Jul 28 17:31:59.609599 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.609578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.612439 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.612420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 17:31:59.612526 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.612516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.612673 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.612656    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 17:31:59.615272 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.615254    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 17:31:59.621823 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.621805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"]
Jul 28 17:31:59.624753 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.624733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"]
Jul 28 17:31:59.629506 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.629487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:59.683340 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdff\" (UniqueName: \"kubernetes.io/projected/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-api-access-rzdff\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.683475 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.683475 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.683603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.683603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.683603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.683603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nq5t\" (UniqueName: \"kubernetes.io/projected/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-api-access-5nq5t\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.683793 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.683793 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.683793 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.683669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.784656 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdff\" (UniqueName: \"kubernetes.io/projected/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-api-access-rzdff\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.784656 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.784849 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.784849 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.784849 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.784849 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.784849 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nq5t\" (UniqueName: \"kubernetes.io/projected/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-api-access-5nq5t\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.784849 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:31:59.784865    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.784878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:31:59.784939    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls podName:8bcc9c36-8ebc-43a6-ae49-93b1941b41d9 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:00.284919119 +0000 UTC m=+1275.450554944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls") pod "isvc-canary-create-v2-predictor-5b976d889b-vm6sl" (UID: "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:31:59.784937    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:31:59.785018    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls podName:f0d562e1-6d3b-492f-b27e-0f45cb080c9e nodeName:}" failed. No retries permitted until 2026-07-28 17:32:00.284998984 +0000 UTC m=+1275.450634797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls") pod "isvc-canary-create-predictor-594f5f9bd5-r99jm" (UID: "f0d562e1-6d3b-492f-b27e-0f45cb080c9e") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.785161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.785194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.785186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.785516 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.785447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.785619 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.785601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.785717 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.785697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.785753 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.785714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.793479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.793461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdff\" (UniqueName: \"kubernetes.io/projected/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-api-access-rzdff\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:31:59.793725 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.793704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nq5t\" (UniqueName: \"kubernetes.io/projected/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-api-access-5nq5t\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:31:59.957692 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:31:59.957661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:00.288417 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:00.288316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:32:00.288417 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:00.288372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:32:00.288621 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:32:00.288468    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:32:00.288621 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:32:00.288532    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls podName:8bcc9c36-8ebc-43a6-ae49-93b1941b41d9 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:01.288516586 +0000 UTC m=+1276.454152398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls") pod "isvc-canary-create-v2-predictor-5b976d889b-vm6sl" (UID: "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:32:00.290757 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:00.290741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-r99jm\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:32:00.377580 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:00.377545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:00.521152 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:00.521107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:32:00.655008 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:00.654986    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"]
Jul 28 17:32:00.657185 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:32:00.657161    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d562e1_6d3b_492f_b27e_0f45cb080c9e.slice/crio-b74835a350673ead86db13fddbd842991c5ddc4c5d36aa37d06f17ac2e4be762 WatchSource:0}: Error finding container b74835a350673ead86db13fddbd842991c5ddc4c5d36aa37d06f17ac2e4be762: Status 404 returned error can't find the container with id b74835a350673ead86db13fddbd842991c5ddc4c5d36aa37d06f17ac2e4be762
Jul 28 17:32:01.297549 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:01.297514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:32:01.300013 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:01.299990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-vm6sl\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:32:01.388792 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:01.388747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerStarted","Data":"6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2"}
Jul 28 17:32:01.388964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:01.388815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerStarted","Data":"b74835a350673ead86db13fddbd842991c5ddc4c5d36aa37d06f17ac2e4be762"}
Jul 28 17:32:01.427626 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:01.427595    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:32:01.555156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:01.555011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"]
Jul 28 17:32:01.557347 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:32:01.557321    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcc9c36_8ebc_43a6_ae49_93b1941b41d9.slice/crio-d3f46c37f0a8d9e9e5f3023b79d602ed92c978a88c7e686b731d9406a0a2480e WatchSource:0}: Error finding container d3f46c37f0a8d9e9e5f3023b79d602ed92c978a88c7e686b731d9406a0a2480e: Status 404 returned error can't find the container with id d3f46c37f0a8d9e9e5f3023b79d602ed92c978a88c7e686b731d9406a0a2480e
Jul 28 17:32:02.394734 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:02.394691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerStarted","Data":"808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f"}
Jul 28 17:32:02.395109 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:02.394772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerStarted","Data":"d3f46c37f0a8d9e9e5f3023b79d602ed92c978a88c7e686b731d9406a0a2480e"}
Jul 28 17:32:02.785841 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:02.785748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 17:32:04.178312 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.178276    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:32:04.217882 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.217854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls\") pod \"1c0e2d7d-7ded-49f8-b634-163710f392d7\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") "
Jul 28 17:32:04.218028 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.217899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-api-access-h2d47\") pod \"1c0e2d7d-7ded-49f8-b634-163710f392d7\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") "
Jul 28 17:32:04.218028 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.217947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e2d7d-7ded-49f8-b634-163710f392d7-kserve-provision-location\") pod \"1c0e2d7d-7ded-49f8-b634-163710f392d7\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") "
Jul 28 17:32:04.218028 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.217971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-cabundle-cert\") pod \"1c0e2d7d-7ded-49f8-b634-163710f392d7\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") "
Jul 28 17:32:04.218028 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.217992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-rbac-proxy-sar-config\") pod \"1c0e2d7d-7ded-49f8-b634-163710f392d7\" (UID: \"1c0e2d7d-7ded-49f8-b634-163710f392d7\") "
Jul 28 17:32:04.218341 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.218294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e2d7d-7ded-49f8-b634-163710f392d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c0e2d7d-7ded-49f8-b634-163710f392d7" (UID: "1c0e2d7d-7ded-49f8-b634-163710f392d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:04.218480 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.218458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c0e2d7d-7ded-49f8-b634-163710f392d7" (UID: "1c0e2d7d-7ded-49f8-b634-163710f392d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:04.218537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.218480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c0e2d7d-7ded-49f8-b634-163710f392d7" (UID: "1c0e2d7d-7ded-49f8-b634-163710f392d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:04.219979 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.219961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c0e2d7d-7ded-49f8-b634-163710f392d7" (UID: "1c0e2d7d-7ded-49f8-b634-163710f392d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:04.220044 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.220030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-api-access-h2d47" (OuterVolumeSpecName: "kube-api-access-h2d47") pod "1c0e2d7d-7ded-49f8-b634-163710f392d7" (UID: "1c0e2d7d-7ded-49f8-b634-163710f392d7"). InnerVolumeSpecName "kube-api-access-h2d47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:04.318647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.318573    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e2d7d-7ded-49f8-b634-163710f392d7-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.318647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.318599    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.318647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.318611    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.318647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.318619    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c0e2d7d-7ded-49f8-b634-163710f392d7-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.318647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.318629    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/1c0e2d7d-7ded-49f8-b634-163710f392d7-kube-api-access-h2d47\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.405088 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.405053    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerID="541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5" exitCode=0
Jul 28 17:32:04.405239 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.405110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"
Jul 28 17:32:04.405239 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.405119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerDied","Data":"541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5"}
Jul 28 17:32:04.405239 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.405181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n" event={"ID":"1c0e2d7d-7ded-49f8-b634-163710f392d7","Type":"ContainerDied","Data":"49ce532791ef7762465de982ae03b8c080d0d17e8fe88796483ebbcf4f5523b9"}
Jul 28 17:32:04.405239 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.405217    2578 scope.go:117] "RemoveContainer" containerID="c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2"
Jul 28 17:32:04.413485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.413470    2578 scope.go:117] "RemoveContainer" containerID="541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5"
Jul 28 17:32:04.420490 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.420474    2578 scope.go:117] "RemoveContainer" containerID="2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55"
Jul 28 17:32:04.427969 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.427954    2578 scope.go:117] "RemoveContainer" containerID="c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2"
Jul 28 17:32:04.428034 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.427993    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"]
Jul 28 17:32:04.428230 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:32:04.428211    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2\": container with ID starting with c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2 not found: ID does not exist" containerID="c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2"
Jul 28 17:32:04.428289 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.428236    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2"} err="failed to get container status \"c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2\": rpc error: code = NotFound desc = could not find container \"c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2\": container with ID starting with c9a2238333d46208f90aacd4cdfceebe3e84e75f83c540ee3b9087c5db8f60c2 not found: ID does not exist"
Jul 28 17:32:04.428289 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.428252    2578 scope.go:117] "RemoveContainer" containerID="541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5"
Jul 28 17:32:04.428481 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:32:04.428467    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5\": container with ID starting with 541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5 not found: ID does not exist" containerID="541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5"
Jul 28 17:32:04.428525 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.428483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5"} err="failed to get container status \"541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5\": rpc error: code = NotFound desc = could not find container \"541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5\": container with ID starting with 541249ddf1b6ee1ff61435a5d6f548865d30cbf000c7638b70fe2d44dc496cf5 not found: ID does not exist"
Jul 28 17:32:04.428525 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.428495    2578 scope.go:117] "RemoveContainer" containerID="2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55"
Jul 28 17:32:04.428703 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:32:04.428684    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55\": container with ID starting with 2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55 not found: ID does not exist" containerID="2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55"
Jul 28 17:32:04.428758 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.428710    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55"} err="failed to get container status \"2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55\": rpc error: code = NotFound desc = could not find container \"2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55\": container with ID starting with 2adc70b9773732d5aec0ceae75113742a009a8a7467909c4f3906a91627cab55 not found: ID does not exist"
Jul 28 17:32:04.431904 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:04.431884    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ffe87-predictor-5f57c755d6-xcr7n"]
Jul 28 17:32:05.363553 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:05.363523    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" path="/var/lib/kubelet/pods/1c0e2d7d-7ded-49f8-b634-163710f392d7/volumes"
Jul 28 17:32:05.412342 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:05.412294    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerID="6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2" exitCode=0
Jul 28 17:32:05.412507 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:05.412348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerDied","Data":"6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2"}
Jul 28 17:32:06.418473 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.418434    2578 generic.go:356] "Generic (PLEG): container finished" podID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerID="808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f" exitCode=0
Jul 28 17:32:06.418882 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.418504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerDied","Data":"808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f"}
Jul 28 17:32:06.420807 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.420784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerStarted","Data":"63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61"}
Jul 28 17:32:06.420886 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.420847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerStarted","Data":"519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8"}
Jul 28 17:32:06.421084 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.421063    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:32:06.421159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.421105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:32:06.422382 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.422358    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:06.461765 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:06.461724    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podStartSLOduration=3.636887969 podStartE2EDuration="7.461708814s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824820845s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:06.459852635 +0000 UTC m=+1281.625488469" watchObservedRunningTime="2026-07-28 17:32:06.461708814 +0000 UTC m=+1281.627344649"
Jul 28 17:32:07.426873 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:07.426841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerStarted","Data":"f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80"}
Jul 28 17:32:07.427345 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:07.426909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerStarted","Data":"9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41"}
Jul 28 17:32:07.427345 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:07.427056    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:07.427345 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:07.427146    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:32:07.427345 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:07.427187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:32:07.428586 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:07.428562    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:07.451745 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:07.451702    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podStartSLOduration=4.579907702 podStartE2EDuration="8.451688798s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.871781096s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:07.449636466 +0000 UTC m=+1282.615272324" watchObservedRunningTime="2026-07-28 17:32:07.451688798 +0000 UTC m=+1282.617324632"
Jul 28 17:32:08.431223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:08.431181    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:12.432244 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:12.432212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:32:12.432797 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:12.432738    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:13.435817 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:13.435786    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:32:13.436400 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:13.436372    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:22.433013 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:22.432970    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:23.437070 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:23.437024    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:32.433584 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:32.433545    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:33.436419 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:33.436379    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:42.432775 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:42.432730    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:43.436243 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:43.436200    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:52.433231 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:52.433193    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:53.437140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:32:53.437098    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:02.433383 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:02.433340    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:03.436462 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:03.436418    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:12.432940 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:12.432860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:13.436975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:13.436937    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:13.583988 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:13.583957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:14.358622 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:14.358583    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:33:23.436435 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:23.436408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:33:24.359312 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:24.359283    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:33:29.692707 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.692660    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"]
Jul 28 17:33:29.693173 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.692953    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" containerID="cri-o://9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41" gracePeriod=30
Jul 28 17:33:29.693374 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.693316    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kube-rbac-proxy" containerID="cri-o://f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80" gracePeriod=30
Jul 28 17:33:29.694360 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.694338    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"]
Jul 28 17:33:29.694587 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.694559    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" containerID="cri-o://519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8" gracePeriod=30
Jul 28 17:33:29.694710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.694571    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kube-rbac-proxy" containerID="cri-o://63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61" gracePeriod=30
Jul 28 17:33:29.721492 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.721464    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:29.777081 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:29.777053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:30.745061 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:30.745031    2578 generic.go:356] "Generic (PLEG): container finished" podID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerID="f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80" exitCode=2
Jul 28 17:33:30.745485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:30.745106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerDied","Data":"f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80"}
Jul 28 17:33:30.747086 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:30.747066    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerID="63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61" exitCode=2
Jul 28 17:33:30.747208 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:30.747133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerDied","Data":"63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61"}
Jul 28 17:33:32.428196 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:32.428155    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 17:33:33.432452 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:33.432412    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 17:33:33.436737 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:33.436716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:34.363595 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.363569    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:33:34.366233 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.366213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:33:34.552539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-rbac-proxy-sar-config\") pod \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") "
Jul 28 17:33:34.552539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552544    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls\") pod \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552570    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-rbac-proxy-sar-config\") pod \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kserve-provision-location\") pod \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552676    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kserve-provision-location\") pod \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nq5t\" (UniqueName: \"kubernetes.io/projected/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-api-access-5nq5t\") pod \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls\") pod \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-cabundle-cert\") pod \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\" (UID: \"f0d562e1-6d3b-492f-b27e-0f45cb080c9e\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-cabundle-cert\") pod \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzdff\" (UniqueName: \"kubernetes.io/projected/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-api-access-rzdff\") pod \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\" (UID: \"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9\") "
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0d562e1-6d3b-492f-b27e-0f45cb080c9e" (UID: "f0d562e1-6d3b-492f-b27e-0f45cb080c9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.552964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" (UID: "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:34.553037 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.553027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" (UID: "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:34.553615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.553114    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.553615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.553132    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.553615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.553145    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.553615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.553327    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" (UID: "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:34.553615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.553425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0d562e1-6d3b-492f-b27e-0f45cb080c9e" (UID: "f0d562e1-6d3b-492f-b27e-0f45cb080c9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:34.553615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.553451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0d562e1-6d3b-492f-b27e-0f45cb080c9e" (UID: "f0d562e1-6d3b-492f-b27e-0f45cb080c9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:34.554844 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.554820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0d562e1-6d3b-492f-b27e-0f45cb080c9e" (UID: "f0d562e1-6d3b-492f-b27e-0f45cb080c9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:34.554844 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.554832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-api-access-5nq5t" (OuterVolumeSpecName: "kube-api-access-5nq5t") pod "f0d562e1-6d3b-492f-b27e-0f45cb080c9e" (UID: "f0d562e1-6d3b-492f-b27e-0f45cb080c9e"). InnerVolumeSpecName "kube-api-access-5nq5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:34.554987 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.554900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-api-access-rzdff" (OuterVolumeSpecName: "kube-api-access-rzdff") pod "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" (UID: "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9"). InnerVolumeSpecName "kube-api-access-rzdff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:34.554987 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.554979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" (UID: "8bcc9c36-8ebc-43a6-ae49-93b1941b41d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:34.654342 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.654258    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nq5t\" (UniqueName: \"kubernetes.io/projected/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-api-access-5nq5t\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.654342 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.654282    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.654342 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.654293    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.654342 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.654319    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.654565 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.654350    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdff\" (UniqueName: \"kubernetes.io/projected/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9-kube-api-access-rzdff\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.654565 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.654358    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.654565 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.654370    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0d562e1-6d3b-492f-b27e-0f45cb080c9e-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:33:34.763548 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.763517    2578 generic.go:356] "Generic (PLEG): container finished" podID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerID="9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41" exitCode=0
Jul 28 17:33:34.763699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.763572    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"
Jul 28 17:33:34.763699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.763602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerDied","Data":"9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41"}
Jul 28 17:33:34.763699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.763666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl" event={"ID":"8bcc9c36-8ebc-43a6-ae49-93b1941b41d9","Type":"ContainerDied","Data":"d3f46c37f0a8d9e9e5f3023b79d602ed92c978a88c7e686b731d9406a0a2480e"}
Jul 28 17:33:34.763699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.763684    2578 scope.go:117] "RemoveContainer" containerID="f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80"
Jul 28 17:33:34.765530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.765507    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerID="519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8" exitCode=0
Jul 28 17:33:34.765653 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.765558    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"
Jul 28 17:33:34.765653 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.765584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerDied","Data":"519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8"}
Jul 28 17:33:34.765731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.765649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" event={"ID":"f0d562e1-6d3b-492f-b27e-0f45cb080c9e","Type":"ContainerDied","Data":"b74835a350673ead86db13fddbd842991c5ddc4c5d36aa37d06f17ac2e4be762"}
Jul 28 17:33:34.772922 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.772872    2578 scope.go:117] "RemoveContainer" containerID="9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41"
Jul 28 17:33:34.781280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.781257    2578 scope.go:117] "RemoveContainer" containerID="808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f"
Jul 28 17:33:34.788690 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.788672    2578 scope.go:117] "RemoveContainer" containerID="f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80"
Jul 28 17:33:34.788966 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:34.788949    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80\": container with ID starting with f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80 not found: ID does not exist" containerID="f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80"
Jul 28 17:33:34.789018 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.788976    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80"} err="failed to get container status \"f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80\": rpc error: code = NotFound desc = could not find container \"f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80\": container with ID starting with f6c47a330d467714ec6d1ab61f3942ead3d0abd39fc96718cf63a0211c4b4c80 not found: ID does not exist"
Jul 28 17:33:34.789018 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.788992    2578 scope.go:117] "RemoveContainer" containerID="9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41"
Jul 28 17:33:34.789214 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:34.789195    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41\": container with ID starting with 9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41 not found: ID does not exist" containerID="9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41"
Jul 28 17:33:34.789285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.789222    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41"} err="failed to get container status \"9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41\": rpc error: code = NotFound desc = could not find container \"9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41\": container with ID starting with 9ac63f4ac186d22bdc9186b0ad6db360f685db4bed4d342aaaf658ca83e50d41 not found: ID does not exist"
Jul 28 17:33:34.789285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.789242    2578 scope.go:117] "RemoveContainer" containerID="808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f"
Jul 28 17:33:34.789513 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:34.789492    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f\": container with ID starting with 808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f not found: ID does not exist" containerID="808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f"
Jul 28 17:33:34.789553 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.789522    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f"} err="failed to get container status \"808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f\": rpc error: code = NotFound desc = could not find container \"808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f\": container with ID starting with 808e466a41224fca1c92b8718da71c40d563ff899526d99bd6f71b2de7767c0f not found: ID does not exist"
Jul 28 17:33:34.789553 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.789543    2578 scope.go:117] "RemoveContainer" containerID="63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61"
Jul 28 17:33:34.796634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.796621    2578 scope.go:117] "RemoveContainer" containerID="519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8"
Jul 28 17:33:34.801651 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.801633    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"]
Jul 28 17:33:34.803646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.803631    2578 scope.go:117] "RemoveContainer" containerID="6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2"
Jul 28 17:33:34.809386 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.809367    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm"]
Jul 28 17:33:34.810975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.810956    2578 scope.go:117] "RemoveContainer" containerID="63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61"
Jul 28 17:33:34.811188 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:34.811173    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61\": container with ID starting with 63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61 not found: ID does not exist" containerID="63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61"
Jul 28 17:33:34.811254 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.811191    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61"} err="failed to get container status \"63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61\": rpc error: code = NotFound desc = could not find container \"63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61\": container with ID starting with 63e2b16b3711af56841a1db088b5248a1699abd0f0f81633d29501e8399dac61 not found: ID does not exist"
Jul 28 17:33:34.811254 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.811205    2578 scope.go:117] "RemoveContainer" containerID="519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8"
Jul 28 17:33:34.811454 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:34.811438    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8\": container with ID starting with 519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8 not found: ID does not exist" containerID="519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8"
Jul 28 17:33:34.811502 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.811457    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8"} err="failed to get container status \"519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8\": rpc error: code = NotFound desc = could not find container \"519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8\": container with ID starting with 519342a6e4e54cd2f7811c7463d5c3e778bfc4009d75506a15cbe42b899d96c8 not found: ID does not exist"
Jul 28 17:33:34.811502 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.811469    2578 scope.go:117] "RemoveContainer" containerID="6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2"
Jul 28 17:33:34.811715 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:34.811684    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2\": container with ID starting with 6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2 not found: ID does not exist" containerID="6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2"
Jul 28 17:33:34.811761 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.811719    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2"} err="failed to get container status \"6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2\": rpc error: code = NotFound desc = could not find container \"6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2\": container with ID starting with 6153ee08654775f23002c6ff9dff7ec39dfd978d3d2ff42797683bb79605fce2 not found: ID does not exist"
Jul 28 17:33:34.814878 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.814856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:34.826992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.826972    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"]
Jul 28 17:33:34.833475 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.833453    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-vm6sl"]
Jul 28 17:33:34.899606 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.899583    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:34.903070 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903049    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"]
Jul 28 17:33:34.903349 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903335    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="storage-initializer"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903352    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="storage-initializer"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903360    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903366    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903378    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903383    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903390    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903411    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903420    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903426    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container"
Jul 28 17:33:34.903427 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903431    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903436    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903445    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903450    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903458    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="storage-initializer"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="storage-initializer"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903470    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="storage-initializer"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903475    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="storage-initializer"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903513    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903521    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kserve-container"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903527    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kube-rbac-proxy"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903538    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0e2d7d-7ded-49f8-b634-163710f392d7" containerName="kserve-container"
Jul 28 17:33:34.903759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.903542    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" containerName="kube-rbac-proxy"
Jul 28 17:33:34.908268 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.908222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:34.908370 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.908294    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"]
Jul 28 17:33:34.911213 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 17:33:34.911346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911195    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:33:34.911346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911236    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:33:34.911498 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:34.911568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911526    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:33:34.911614 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w2njn\""
Jul 28 17:33:34.911659 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:33:34.911659 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.911614    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:33:34.914183 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.914166    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:33:34.919070 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.919043    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"]
Jul 28 17:33:34.922673 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:34.922652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"]
Jul 28 17:33:35.058180 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6863d7a2-ef6d-4317-864b-b137906128f2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.058180 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.058414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.058414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.058414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.058414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.058414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28srf\" (UniqueName: \"kubernetes.io/projected/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-api-access-28srf\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.058573 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.058573 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f5f\" (UniqueName: \"kubernetes.io/projected/6863d7a2-ef6d-4317-864b-b137906128f2-kube-api-access-r4f5f\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.058573 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.058499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.158868 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.158806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6863d7a2-ef6d-4317-864b-b137906128f2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.158868 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.158840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.158868 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.158858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.158881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.158907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.159092 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:35.158983    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 17:33:35.159092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.159092 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:35.159055    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls podName:6863d7a2-ef6d-4317-864b-b137906128f2 nodeName:}" failed. No retries permitted until 2026-07-28 17:33:35.659032811 +0000 UTC m=+1370.824668625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls") pod "isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" (UID: "6863d7a2-ef6d-4317-864b-b137906128f2") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 17:33:35.159402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28srf\" (UniqueName: \"kubernetes.io/projected/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-api-access-28srf\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f5f\" (UniqueName: \"kubernetes.io/projected/6863d7a2-ef6d-4317-864b-b137906128f2-kube-api-access-r4f5f\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.159402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6863d7a2-ef6d-4317-864b-b137906128f2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.159402 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:35.159373    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:33:35.159715 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:33:35.159421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls podName:d51a26c8-4582-44ee-87fd-0fd66e3c9760 nodeName:}" failed. No retries permitted until 2026-07-28 17:33:35.659405277 +0000 UTC m=+1370.825041090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls") pod "isvc-canary-promote-predictor-5d7dcd97d5-9dccm" (UID: "d51a26c8-4582-44ee-87fd-0fd66e3c9760") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:33:35.159715 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159715 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159843 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.159843 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.159843 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.159774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.168392 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.168373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f5f\" (UniqueName: \"kubernetes.io/projected/6863d7a2-ef6d-4317-864b-b137906128f2-kube-api-access-r4f5f\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.168493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.168413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28srf\" (UniqueName: \"kubernetes.io/projected/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-api-access-28srf\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.272810 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.272776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:35.359924 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.359889    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-r99jm" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: i/o timeout"
Jul 28 17:33:35.364479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.364455    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bcc9c36-8ebc-43a6-ae49-93b1941b41d9" path="/var/lib/kubelet/pods/8bcc9c36-8ebc-43a6-ae49-93b1941b41d9/volumes"
Jul 28 17:33:35.364976 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.364962    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d562e1-6d3b-492f-b27e-0f45cb080c9e" path="/var/lib/kubelet/pods/f0d562e1-6d3b-492f-b27e-0f45cb080c9e/volumes"
Jul 28 17:33:35.372941 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.372921    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:35.663307 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.663276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.663760 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.663343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.665646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.665621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-9dccm\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.665748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.665668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ph2v7\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.820115 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.820088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:35.827160 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.827145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:35.955477 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.955439    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"]
Jul 28 17:33:35.958677 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:33:35.958645    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51a26c8_4582_44ee_87fd_0fd66e3c9760.slice/crio-49bf81a8a7fff2bc5187b8a554924ec8f0f8d759f4e1d9ab86b610ff3e55508c WatchSource:0}: Error finding container 49bf81a8a7fff2bc5187b8a554924ec8f0f8d759f4e1d9ab86b610ff3e55508c: Status 404 returned error can't find the container with id 49bf81a8a7fff2bc5187b8a554924ec8f0f8d759f4e1d9ab86b610ff3e55508c
Jul 28 17:33:35.972620 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:35.972598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"]
Jul 28 17:33:35.974501 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:33:35.974476    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6863d7a2_ef6d_4317_864b_b137906128f2.slice/crio-bfba4fd4a78c545c9b12a94c917aacb2774c6d60ceaa843001d96f3917d67248 WatchSource:0}: Error finding container bfba4fd4a78c545c9b12a94c917aacb2774c6d60ceaa843001d96f3917d67248: Status 404 returned error can't find the container with id bfba4fd4a78c545c9b12a94c917aacb2774c6d60ceaa843001d96f3917d67248
Jul 28 17:33:36.775807 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:36.775773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerStarted","Data":"dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35"}
Jul 28 17:33:36.776256 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:36.775843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerStarted","Data":"49bf81a8a7fff2bc5187b8a554924ec8f0f8d759f4e1d9ab86b610ff3e55508c"}
Jul 28 17:33:36.777387 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:36.777366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerStarted","Data":"20da5f6630583d6cfd4386c090cd0e497b188f06fb59d5235e9e7eab3ae8ef99"}
Jul 28 17:33:36.777441 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:36.777410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerStarted","Data":"bfba4fd4a78c545c9b12a94c917aacb2774c6d60ceaa843001d96f3917d67248"}
Jul 28 17:33:39.789783 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:39.789750    2578 generic.go:356] "Generic (PLEG): container finished" podID="6863d7a2-ef6d-4317-864b-b137906128f2" containerID="20da5f6630583d6cfd4386c090cd0e497b188f06fb59d5235e9e7eab3ae8ef99" exitCode=0
Jul 28 17:33:39.790169 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:39.789823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerDied","Data":"20da5f6630583d6cfd4386c090cd0e497b188f06fb59d5235e9e7eab3ae8ef99"}
Jul 28 17:33:39.791519 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:39.791500    2578 generic.go:356] "Generic (PLEG): container finished" podID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerID="dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35" exitCode=0
Jul 28 17:33:39.791616 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:39.791528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerDied","Data":"dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35"}
Jul 28 17:33:40.799213 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.799177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerStarted","Data":"b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7"}
Jul 28 17:33:40.799696 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.799244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerStarted","Data":"d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a"}
Jul 28 17:33:40.799696 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.799483    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:40.801387 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.801365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerStarted","Data":"123b8f8dfad774f2a40d54b49079edf5ffabb908029d1667e03f8b809855dd63"}
Jul 28 17:33:40.801512 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.801416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerStarted","Data":"6344da22069e7634fbdf2f1f55a3ca2e47a09dca1f09ab947184ee2cfb89ead3"}
Jul 28 17:33:40.801647 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.801627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:40.801767 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.801671    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:40.803226 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.803195    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:33:40.819261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.819219    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podStartSLOduration=3.167040545 podStartE2EDuration="6.819205929s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.652165384s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:40.817759833 +0000 UTC m=+1375.983395678" watchObservedRunningTime="2026-07-28 17:33:40.819205929 +0000 UTC m=+1375.984841763"
Jul 28 17:33:40.837728 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:40.837689    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podStartSLOduration=3.17839676 podStartE2EDuration="6.837676049s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659279289s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:40.836387865 +0000 UTC m=+1376.002023689" watchObservedRunningTime="2026-07-28 17:33:40.837676049 +0000 UTC m=+1376.003311883"
Jul 28 17:33:41.806334 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:41.806275    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:33:41.806765 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:41.806561    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:41.807762 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:41.807735    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:33:42.810765 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:42.810728    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:33:46.810372 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:46.810342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:33:46.810938 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:46.810911    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:33:47.815931 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:47.815905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:33:47.816406 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:47.816380    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:33:56.811093 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:56.811050    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:33:57.816542 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:33:57.816501    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:06.810896 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:06.810860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:34:07.816754 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:07.816712    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:16.810914 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:16.810874    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:34:17.817171 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:17.817132    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:26.811235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:26.811192    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:34:27.816839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:27.816802    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:36.811888 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:36.811849    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:34:37.816569 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:37.816533    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:46.811314 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:46.811263    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:34:47.816607 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:47.816568    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:34:56.812169 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:56.812140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:34:57.817446 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:34:57.817410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:35:04.942982 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:04.942951    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"]
Jul 28 17:35:04.943401 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:04.943186    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" containerID="cri-o://d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a" gracePeriod=30
Jul 28 17:35:04.943401 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:04.943225    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kube-rbac-proxy" containerID="cri-o://b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7" gracePeriod=30
Jul 28 17:35:04.969265 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:04.969239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:05.120850 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:05.120817    2578 generic.go:356] "Generic (PLEG): container finished" podID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerID="b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7" exitCode=2
Jul 28 17:35:05.120991 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:05.120864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerDied","Data":"b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7"}
Jul 28 17:35:07.811135 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:07.811081    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 17:35:07.816893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:07.816866    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:35:09.385133 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.385110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:35:09.539828 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.539734    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-cabundle-cert\") pod \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") "
Jul 28 17:35:09.539828 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.539780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-rbac-proxy-sar-config\") pod \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") "
Jul 28 17:35:09.540026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.539909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28srf\" (UniqueName: \"kubernetes.io/projected/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-api-access-28srf\") pod \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") "
Jul 28 17:35:09.540026 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.539998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kserve-provision-location\") pod \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") "
Jul 28 17:35:09.540125 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.540056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls\") pod \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\" (UID: \"d51a26c8-4582-44ee-87fd-0fd66e3c9760\") "
Jul 28 17:35:09.540184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.540152    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d51a26c8-4582-44ee-87fd-0fd66e3c9760" (UID: "d51a26c8-4582-44ee-87fd-0fd66e3c9760"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:09.540184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.540157    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d51a26c8-4582-44ee-87fd-0fd66e3c9760" (UID: "d51a26c8-4582-44ee-87fd-0fd66e3c9760"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:09.540361 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.540342    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:35:09.540450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.540364    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:35:09.540450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.540386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d51a26c8-4582-44ee-87fd-0fd66e3c9760" (UID: "d51a26c8-4582-44ee-87fd-0fd66e3c9760"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:35:09.542092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.542061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-api-access-28srf" (OuterVolumeSpecName: "kube-api-access-28srf") pod "d51a26c8-4582-44ee-87fd-0fd66e3c9760" (UID: "d51a26c8-4582-44ee-87fd-0fd66e3c9760"). InnerVolumeSpecName "kube-api-access-28srf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:09.542229 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.542152    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d51a26c8-4582-44ee-87fd-0fd66e3c9760" (UID: "d51a26c8-4582-44ee-87fd-0fd66e3c9760"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:09.640821 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.640795    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28srf\" (UniqueName: \"kubernetes.io/projected/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kube-api-access-28srf\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:35:09.640821 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.640816    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d51a26c8-4582-44ee-87fd-0fd66e3c9760-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:35:09.640972 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:09.640826    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d51a26c8-4582-44ee-87fd-0fd66e3c9760-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:35:10.141729 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.141697    2578 generic.go:356] "Generic (PLEG): container finished" podID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerID="d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a" exitCode=0
Jul 28 17:35:10.141908 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.141754    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"
Jul 28 17:35:10.141908 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.141778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerDied","Data":"d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a"}
Jul 28 17:35:10.141908 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.141851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm" event={"ID":"d51a26c8-4582-44ee-87fd-0fd66e3c9760","Type":"ContainerDied","Data":"49bf81a8a7fff2bc5187b8a554924ec8f0f8d759f4e1d9ab86b610ff3e55508c"}
Jul 28 17:35:10.141908 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.141890    2578 scope.go:117] "RemoveContainer" containerID="b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7"
Jul 28 17:35:10.150122 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.150108    2578 scope.go:117] "RemoveContainer" containerID="d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a"
Jul 28 17:35:10.156949 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.156935    2578 scope.go:117] "RemoveContainer" containerID="dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35"
Jul 28 17:35:10.163172 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.163151    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"]
Jul 28 17:35:10.164078 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.164061    2578 scope.go:117] "RemoveContainer" containerID="b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7"
Jul 28 17:35:10.164347 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:35:10.164320    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7\": container with ID starting with b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7 not found: ID does not exist" containerID="b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7"
Jul 28 17:35:10.164426 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.164357    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7"} err="failed to get container status \"b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7\": rpc error: code = NotFound desc = could not find container \"b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7\": container with ID starting with b781be7ed4395feac32b451bf206fc9739cf1c0d2ef45d365efe85a11acc8da7 not found: ID does not exist"
Jul 28 17:35:10.164426 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.164376    2578 scope.go:117] "RemoveContainer" containerID="d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a"
Jul 28 17:35:10.164647 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:35:10.164626    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a\": container with ID starting with d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a not found: ID does not exist" containerID="d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a"
Jul 28 17:35:10.164742 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.164650    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a"} err="failed to get container status \"d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a\": rpc error: code = NotFound desc = could not find container \"d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a\": container with ID starting with d1aabb2f01cbbc7a8f8448da40fa8098d63db83b117fc3411b1e06d90239cb1a not found: ID does not exist"
Jul 28 17:35:10.164742 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.164667    2578 scope.go:117] "RemoveContainer" containerID="dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35"
Jul 28 17:35:10.165002 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:35:10.164984    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35\": container with ID starting with dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35 not found: ID does not exist" containerID="dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35"
Jul 28 17:35:10.165076 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.165005    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35"} err="failed to get container status \"dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35\": rpc error: code = NotFound desc = could not find container \"dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35\": container with ID starting with dd15e73329d319bb0faf4c59adb162c249e11a8f8cc8fba173b8be9a13d2ae35 not found: ID does not exist"
Jul 28 17:35:10.167147 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:10.167127    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-9dccm"]
Jul 28 17:35:11.362916 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:11.362886    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" path="/var/lib/kubelet/pods/d51a26c8-4582-44ee-87fd-0fd66e3c9760/volumes"
Jul 28 17:35:35.563208 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:35:35.563175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:45.195434 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:45.195399    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"]
Jul 28 17:36:45.196001 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:45.195691    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" containerID="cri-o://6344da22069e7634fbdf2f1f55a3ca2e47a09dca1f09ab947184ee2cfb89ead3" gracePeriod=30
Jul 28 17:36:45.196001 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:45.195748    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kube-rbac-proxy" containerID="cri-o://123b8f8dfad774f2a40d54b49079edf5ffabb908029d1667e03f8b809855dd63" gracePeriod=30
Jul 28 17:36:45.223756 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:45.223727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:45.491364 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:45.491270    2578 generic.go:356] "Generic (PLEG): container finished" podID="6863d7a2-ef6d-4317-864b-b137906128f2" containerID="123b8f8dfad774f2a40d54b49079edf5ffabb908029d1667e03f8b809855dd63" exitCode=2
Jul 28 17:36:45.491364 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:45.491336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerDied","Data":"123b8f8dfad774f2a40d54b49079edf5ffabb908029d1667e03f8b809855dd63"}
Jul 28 17:36:46.807368 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:46.807326    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 17:36:46.811569 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:46.811536    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:36:49.509350 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.509292    2578 generic.go:356] "Generic (PLEG): container finished" podID="6863d7a2-ef6d-4317-864b-b137906128f2" containerID="6344da22069e7634fbdf2f1f55a3ca2e47a09dca1f09ab947184ee2cfb89ead3" exitCode=0
Jul 28 17:36:49.509350 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.509335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerDied","Data":"6344da22069e7634fbdf2f1f55a3ca2e47a09dca1f09ab947184ee2cfb89ead3"}
Jul 28 17:36:49.536173 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.536151    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:36:49.637625 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.637593    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6863d7a2-ef6d-4317-864b-b137906128f2-kserve-provision-location\") pod \"6863d7a2-ef6d-4317-864b-b137906128f2\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") "
Jul 28 17:36:49.637625 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.637632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls\") pod \"6863d7a2-ef6d-4317-864b-b137906128f2\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") "
Jul 28 17:36:49.637822 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.637651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-kube-rbac-proxy-sar-config\") pod \"6863d7a2-ef6d-4317-864b-b137906128f2\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") "
Jul 28 17:36:49.637822 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.637696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4f5f\" (UniqueName: \"kubernetes.io/projected/6863d7a2-ef6d-4317-864b-b137906128f2-kube-api-access-r4f5f\") pod \"6863d7a2-ef6d-4317-864b-b137906128f2\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") "
Jul 28 17:36:49.637822 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.637729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-cabundle-cert\") pod \"6863d7a2-ef6d-4317-864b-b137906128f2\" (UID: \"6863d7a2-ef6d-4317-864b-b137906128f2\") "
Jul 28 17:36:49.637940 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.637900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6863d7a2-ef6d-4317-864b-b137906128f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6863d7a2-ef6d-4317-864b-b137906128f2" (UID: "6863d7a2-ef6d-4317-864b-b137906128f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:49.638124 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.638096    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6863d7a2-ef6d-4317-864b-b137906128f2" (UID: "6863d7a2-ef6d-4317-864b-b137906128f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:49.638187 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.638107    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6863d7a2-ef6d-4317-864b-b137906128f2" (UID: "6863d7a2-ef6d-4317-864b-b137906128f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:49.639670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.639649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6863d7a2-ef6d-4317-864b-b137906128f2" (UID: "6863d7a2-ef6d-4317-864b-b137906128f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:49.639782 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.639761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6863d7a2-ef6d-4317-864b-b137906128f2-kube-api-access-r4f5f" (OuterVolumeSpecName: "kube-api-access-r4f5f") pod "6863d7a2-ef6d-4317-864b-b137906128f2" (UID: "6863d7a2-ef6d-4317-864b-b137906128f2"). InnerVolumeSpecName "kube-api-access-r4f5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:49.739148 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.739103    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6863d7a2-ef6d-4317-864b-b137906128f2-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:36:49.739148 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.739143    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6863d7a2-ef6d-4317-864b-b137906128f2-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:36:49.739148 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.739156    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:36:49.739411 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.739165    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f5f\" (UniqueName: \"kubernetes.io/projected/6863d7a2-ef6d-4317-864b-b137906128f2-kube-api-access-r4f5f\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:36:49.739411 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:49.739176    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6863d7a2-ef6d-4317-864b-b137906128f2-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:36:50.275771 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.275730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:50.338199 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338169    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"]
Jul 28 17:36:50.338479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338466    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="storage-initializer"
Jul 28 17:36:50.338530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338481    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="storage-initializer"
Jul 28 17:36:50.338530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kube-rbac-proxy"
Jul 28 17:36:50.338530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338499    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kube-rbac-proxy"
Jul 28 17:36:50.338530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338507    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="storage-initializer"
Jul 28 17:36:50.338530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338514    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="storage-initializer"
Jul 28 17:36:50.338530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338529    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338535    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338544    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338549    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338559    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kube-rbac-proxy"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338564    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kube-rbac-proxy"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338610    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kube-rbac-proxy"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338618    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" containerName="kserve-container"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338626    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kube-rbac-proxy"
Jul 28 17:36:50.339020 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.338632    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51a26c8-4582-44ee-87fd-0fd66e3c9760" containerName="kserve-container"
Jul 28 17:36:50.341673 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.341655    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"]
Jul 28 17:36:50.341812 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.341793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.344684 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.344661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:36:50.344803 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.344666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.344803 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.344730    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:36:50.347223 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.347204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 17:36:50.353556 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.353534    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"]
Jul 28 17:36:50.355585 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.355521    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"]
Jul 28 17:36:50.380479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.380436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:50.445496 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.445617 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.445617 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.445701 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.445701 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55mn\" (UniqueName: \"kubernetes.io/projected/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-api-access-b55mn\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.445701 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3439b76-c6d0-49d4-960d-f3115bb96a70-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.445810 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjl6\" (UniqueName: \"kubernetes.io/projected/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-api-access-jnjl6\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.445810 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.445880 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.445880 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.445850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.515487 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.515459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7" event={"ID":"6863d7a2-ef6d-4317-864b-b137906128f2","Type":"ContainerDied","Data":"bfba4fd4a78c545c9b12a94c917aacb2774c6d60ceaa843001d96f3917d67248"}
Jul 28 17:36:50.515840 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.515548    2578 scope.go:117] "RemoveContainer" containerID="123b8f8dfad774f2a40d54b49079edf5ffabb908029d1667e03f8b809855dd63"
Jul 28 17:36:50.515840 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.515470    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"
Jul 28 17:36:50.523749 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.523729    2578 scope.go:117] "RemoveContainer" containerID="6344da22069e7634fbdf2f1f55a3ca2e47a09dca1f09ab947184ee2cfb89ead3"
Jul 28 17:36:50.530612 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.530598    2578 scope.go:117] "RemoveContainer" containerID="20da5f6630583d6cfd4386c090cd0e497b188f06fb59d5235e9e7eab3ae8ef99"
Jul 28 17:36:50.537392 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.537371    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"]
Jul 28 17:36:50.543065 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.543043    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ph2v7"]
Jul 28 17:36:50.547089 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjl6\" (UniqueName: \"kubernetes.io/projected/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-api-access-jnjl6\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.547151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.547151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.547260 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.547364 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.547364 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.547479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.547479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.547479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55mn\" (UniqueName: \"kubernetes.io/projected/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-api-access-b55mn\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.547635 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3439b76-c6d0-49d4-960d-f3115bb96a70-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.547635 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:36:50.547507    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:36:50.547635 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.547635 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:36:50.547571    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls podName:e3439b76-c6d0-49d4-960d-f3115bb96a70 nodeName:}" failed. No retries permitted until 2026-07-28 17:36:51.047551188 +0000 UTC m=+1566.213187002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls") pod "isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" (UID: "e3439b76-c6d0-49d4-960d-f3115bb96a70") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:36:50.547892 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:36:50.547753    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 17:36:50.547892 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:36:50.547814    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls podName:73464ed8-c6b6-426c-8ae9-c4d0822f30d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:36:51.047796896 +0000 UTC m=+1566.213432711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls") pod "isvc-canary-rollback-predictor-b4645bb5c-7w5kv" (UID: "73464ed8-c6b6-426c-8ae9-c4d0822f30d4") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 17:36:50.547892 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.548003 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3439b76-c6d0-49d4-960d-f3115bb96a70-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.548003 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.548003 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.547987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.548093 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.548040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.556180 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.556148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55mn\" (UniqueName: \"kubernetes.io/projected/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-api-access-b55mn\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:50.556987 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.556964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjl6\" (UniqueName: \"kubernetes.io/projected/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-api-access-jnjl6\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:50.827898 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:50.827866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:51.028942 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.028912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:51.052101 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.052075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:51.052222 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.052119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:51.054472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.054451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-7w5kv\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:51.054559 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.054490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:51.252763 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.252675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:51.260074 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.260046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:51.364964 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.364939    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6863d7a2-ef6d-4317-864b-b137906128f2" path="/var/lib/kubelet/pods/6863d7a2-ef6d-4317-864b-b137906128f2/volumes"
Jul 28 17:36:51.393563 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.393542    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"]
Jul 28 17:36:51.395710 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:36:51.395678    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73464ed8_c6b6_426c_8ae9_c4d0822f30d4.slice/crio-425dca7c78e83c38bff4d0efa72f25b7f6e99acb637bd21492935c72663b8849 WatchSource:0}: Error finding container 425dca7c78e83c38bff4d0efa72f25b7f6e99acb637bd21492935c72663b8849: Status 404 returned error can't find the container with id 425dca7c78e83c38bff4d0efa72f25b7f6e99acb637bd21492935c72663b8849
Jul 28 17:36:51.408319 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.408176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"]
Jul 28 17:36:51.410611 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:36:51.410587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3439b76_c6d0_49d4_960d_f3115bb96a70.slice/crio-c4bbe4a962b11aef8711bcc6e5b89c3e0177f44367bccb258512442fd1a2128e WatchSource:0}: Error finding container c4bbe4a962b11aef8711bcc6e5b89c3e0177f44367bccb258512442fd1a2128e: Status 404 returned error can't find the container with id c4bbe4a962b11aef8711bcc6e5b89c3e0177f44367bccb258512442fd1a2128e
Jul 28 17:36:51.521247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.521139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerStarted","Data":"fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67"}
Jul 28 17:36:51.521247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.521230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerStarted","Data":"425dca7c78e83c38bff4d0efa72f25b7f6e99acb637bd21492935c72663b8849"}
Jul 28 17:36:51.522779 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.522757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerStarted","Data":"01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569"}
Jul 28 17:36:51.522905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:51.522813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerStarted","Data":"c4bbe4a962b11aef8711bcc6e5b89c3e0177f44367bccb258512442fd1a2128e"}
Jul 28 17:36:55.540975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:55.540900    2578 generic.go:356] "Generic (PLEG): container finished" podID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerID="fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67" exitCode=0
Jul 28 17:36:55.541368 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:55.540973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerDied","Data":"fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67"}
Jul 28 17:36:55.542607 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:55.542587    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerID="01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569" exitCode=0
Jul 28 17:36:55.542692 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:55.542625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerDied","Data":"01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569"}
Jul 28 17:36:56.548447 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.548411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerStarted","Data":"64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4"}
Jul 28 17:36:56.548886 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.548489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerStarted","Data":"8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b"}
Jul 28 17:36:56.548886 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.548694    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:56.550524 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.550502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerStarted","Data":"7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb"}
Jul 28 17:36:56.550623 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.550548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerStarted","Data":"8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b"}
Jul 28 17:36:56.550860 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.550845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:56.550907 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.550895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:36:56.551916 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.551892    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:36:56.575780 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.575740    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podStartSLOduration=2.725535634 podStartE2EDuration="6.575727477s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.850191843s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:56.573475603 +0000 UTC m=+1571.739111439" watchObservedRunningTime="2026-07-28 17:36:56.575727477 +0000 UTC m=+1571.741363311"
Jul 28 17:36:56.594980 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:56.594943    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podStartSLOduration=2.857269971 podStartE2EDuration="6.59493112s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.737661149s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:56.594006481 +0000 UTC m=+1571.759642316" watchObservedRunningTime="2026-07-28 17:36:56.59493112 +0000 UTC m=+1571.760566954"
Jul 28 17:36:57.554581 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:57.554543    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:36:57.555076 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:57.554868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:36:57.555923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:57.555897    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:36:58.559439 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:36:58.559402    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:37:02.558852 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:02.558822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:37:02.559578 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:02.559530    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:37:03.563344 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:03.563316    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:37:03.563804 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:03.563776    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:37:12.559485 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:12.559431    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:37:13.563944 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:13.563904    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:37:22.560025 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:22.559976    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:37:23.564112 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:23.564068    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:37:32.559891 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:32.559848    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:37:33.563853 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:33.563806    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:37:42.532581 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:42.532504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:42.559522 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:42.559480    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:37:43.563724 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:43.563685    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:37:52.559636 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:52.559592    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:37:53.563825 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:37:53.563788    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:38:02.560024 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:02.559980    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:38:03.564569 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:03.564527    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:38:12.560263 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:12.560230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:38:13.564425 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:13.564395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:38:20.396145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:20.396104    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"]
Jul 28 17:38:20.396602 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:20.396562    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" containerID="cri-o://8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b" gracePeriod=30
Jul 28 17:38:20.397251 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:20.396791    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kube-rbac-proxy" containerID="cri-o://7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb" gracePeriod=30
Jul 28 17:38:20.410184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:20.410159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:20.873311 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:20.873278    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerID="7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb" exitCode=2
Jul 28 17:38:20.873474 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:20.873336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerDied","Data":"7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb"}
Jul 28 17:38:21.968473 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:21.968439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:22.555525 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:22.555477    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 17:38:22.559874 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:22.559835    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:38:24.737934 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.737912    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:38:24.800610 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.800545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-rbac-proxy-sar-config\") pod \"e3439b76-c6d0-49d4-960d-f3115bb96a70\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") "
Jul 28 17:38:24.800610 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.800579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-cabundle-cert\") pod \"e3439b76-c6d0-49d4-960d-f3115bb96a70\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") "
Jul 28 17:38:24.800815 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.800629    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b55mn\" (UniqueName: \"kubernetes.io/projected/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-api-access-b55mn\") pod \"e3439b76-c6d0-49d4-960d-f3115bb96a70\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") "
Jul 28 17:38:24.800815 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.800722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3439b76-c6d0-49d4-960d-f3115bb96a70-kserve-provision-location\") pod \"e3439b76-c6d0-49d4-960d-f3115bb96a70\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") "
Jul 28 17:38:24.800815 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.800764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls\") pod \"e3439b76-c6d0-49d4-960d-f3115bb96a70\" (UID: \"e3439b76-c6d0-49d4-960d-f3115bb96a70\") "
Jul 28 17:38:24.801055 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.801030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3439b76-c6d0-49d4-960d-f3115bb96a70" (UID: "e3439b76-c6d0-49d4-960d-f3115bb96a70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:24.801055 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.801040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3439b76-c6d0-49d4-960d-f3115bb96a70" (UID: "e3439b76-c6d0-49d4-960d-f3115bb96a70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:24.801055 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.801047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3439b76-c6d0-49d4-960d-f3115bb96a70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3439b76-c6d0-49d4-960d-f3115bb96a70" (UID: "e3439b76-c6d0-49d4-960d-f3115bb96a70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:24.802646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.802627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-api-access-b55mn" (OuterVolumeSpecName: "kube-api-access-b55mn") pod "e3439b76-c6d0-49d4-960d-f3115bb96a70" (UID: "e3439b76-c6d0-49d4-960d-f3115bb96a70"). InnerVolumeSpecName "kube-api-access-b55mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:24.802723 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.802704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3439b76-c6d0-49d4-960d-f3115bb96a70" (UID: "e3439b76-c6d0-49d4-960d-f3115bb96a70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:24.890761 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.890738    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerID="8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b" exitCode=0
Jul 28 17:38:24.890901 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.890771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerDied","Data":"8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b"}
Jul 28 17:38:24.890901 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.890800    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"
Jul 28 17:38:24.890901 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.890849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5" event={"ID":"e3439b76-c6d0-49d4-960d-f3115bb96a70","Type":"ContainerDied","Data":"c4bbe4a962b11aef8711bcc6e5b89c3e0177f44367bccb258512442fd1a2128e"}
Jul 28 17:38:24.890901 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.890888    2578 scope.go:117] "RemoveContainer" containerID="7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb"
Jul 28 17:38:24.898812 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.898797    2578 scope.go:117] "RemoveContainer" containerID="8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b"
Jul 28 17:38:24.901950 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.901931    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:38:24.902038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.901953    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3439b76-c6d0-49d4-960d-f3115bb96a70-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:38:24.902038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.901968    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b55mn\" (UniqueName: \"kubernetes.io/projected/e3439b76-c6d0-49d4-960d-f3115bb96a70-kube-api-access-b55mn\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:38:24.902038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.901980    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3439b76-c6d0-49d4-960d-f3115bb96a70-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:38:24.902038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.901994    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3439b76-c6d0-49d4-960d-f3115bb96a70-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:38:24.905963 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.905947    2578 scope.go:117] "RemoveContainer" containerID="01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569"
Jul 28 17:38:24.913081 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.913065    2578 scope.go:117] "RemoveContainer" containerID="7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb"
Jul 28 17:38:24.913354 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:38:24.913337    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb\": container with ID starting with 7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb not found: ID does not exist" containerID="7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb"
Jul 28 17:38:24.913409 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.913361    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb"} err="failed to get container status \"7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb\": rpc error: code = NotFound desc = could not find container \"7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb\": container with ID starting with 7f281e7608b15103bf3f0e8a5317fba5bb2bbce5584451691647ab5c370849bb not found: ID does not exist"
Jul 28 17:38:24.913409 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.913377    2578 scope.go:117] "RemoveContainer" containerID="8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b"
Jul 28 17:38:24.913605 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:38:24.913586    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b\": container with ID starting with 8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b not found: ID does not exist" containerID="8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b"
Jul 28 17:38:24.913646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.913610    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b"} err="failed to get container status \"8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b\": rpc error: code = NotFound desc = could not find container \"8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b\": container with ID starting with 8418de180f4100a159861c7b00d1bb52b84fcb335d2884a0715f5bd5db6f0b6b not found: ID does not exist"
Jul 28 17:38:24.913646 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.913627    2578 scope.go:117] "RemoveContainer" containerID="01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569"
Jul 28 17:38:24.913781 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.913766    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"]
Jul 28 17:38:24.913860 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:38:24.913845    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569\": container with ID starting with 01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569 not found: ID does not exist" containerID="01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569"
Jul 28 17:38:24.913899 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.913864    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569"} err="failed to get container status \"01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569\": rpc error: code = NotFound desc = could not find container \"01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569\": container with ID starting with 01934958eee424bdbb9be53f70f919e86f2f12ab2ebed5d477249afe45b7b569 not found: ID does not exist"
Jul 28 17:38:24.918045 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:24.918027    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jr8r5"]
Jul 28 17:38:25.362959 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:38:25.362925    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" path="/var/lib/kubelet/pods/e3439b76-c6d0-49d4-960d-f3115bb96a70/volumes"
Jul 28 17:39:03.122634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:03.122577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:03.191875 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:03.191849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:50.598009 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:50.597978    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"]
Jul 28 17:39:50.598686 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:50.598244    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" containerID="cri-o://8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b" gracePeriod=30
Jul 28 17:39:50.598686 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:50.598281    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kube-rbac-proxy" containerID="cri-o://64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4" gracePeriod=30
Jul 28 17:39:50.629382 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:50.629358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:51.205748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:51.205719    2578 generic.go:356] "Generic (PLEG): container finished" podID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerID="64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4" exitCode=2
Jul 28 17:39:51.205910 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:51.205782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerDied","Data":"64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4"}
Jul 28 17:39:53.560171 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:53.560129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 28 17:39:53.564438 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:53.564413    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:39:54.837870 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.837849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:39:54.894010 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.893983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnjl6\" (UniqueName: \"kubernetes.io/projected/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-api-access-jnjl6\") pod \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") "
Jul 28 17:39:54.894129 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.894018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls\") pod \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") "
Jul 28 17:39:54.894129 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.894065    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-rbac-proxy-sar-config\") pod \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") "
Jul 28 17:39:54.894129 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.894106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-cabundle-cert\") pod \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") "
Jul 28 17:39:54.894255 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.894130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kserve-provision-location\") pod \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\" (UID: \"73464ed8-c6b6-426c-8ae9-c4d0822f30d4\") "
Jul 28 17:39:54.900280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.894817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73464ed8-c6b6-426c-8ae9-c4d0822f30d4" (UID: "73464ed8-c6b6-426c-8ae9-c4d0822f30d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:39:54.900280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.894829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73464ed8-c6b6-426c-8ae9-c4d0822f30d4" (UID: "73464ed8-c6b6-426c-8ae9-c4d0822f30d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:54.900280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.894882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73464ed8-c6b6-426c-8ae9-c4d0822f30d4" (UID: "73464ed8-c6b6-426c-8ae9-c4d0822f30d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:39:54.900534 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.900478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-api-access-jnjl6" (OuterVolumeSpecName: "kube-api-access-jnjl6") pod "73464ed8-c6b6-426c-8ae9-c4d0822f30d4" (UID: "73464ed8-c6b6-426c-8ae9-c4d0822f30d4"). InnerVolumeSpecName "kube-api-access-jnjl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:39:54.901042 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.901023    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73464ed8-c6b6-426c-8ae9-c4d0822f30d4" (UID: "73464ed8-c6b6-426c-8ae9-c4d0822f30d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:39:54.995559 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.995497    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:39:54.995559 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.995522    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:39:54.995559 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.995537    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnjl6\" (UniqueName: \"kubernetes.io/projected/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-api-access-jnjl6\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:39:54.995559 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.995550    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:39:54.995743 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:54.995565    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73464ed8-c6b6-426c-8ae9-c4d0822f30d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:39:55.222813 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.222774    2578 generic.go:356] "Generic (PLEG): container finished" podID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerID="8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b" exitCode=0
Jul 28 17:39:55.223002 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.222819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerDied","Data":"8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b"}
Jul 28 17:39:55.223002 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.222842    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"
Jul 28 17:39:55.223002 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.222911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv" event={"ID":"73464ed8-c6b6-426c-8ae9-c4d0822f30d4","Type":"ContainerDied","Data":"425dca7c78e83c38bff4d0efa72f25b7f6e99acb637bd21492935c72663b8849"}
Jul 28 17:39:55.223002 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.222951    2578 scope.go:117] "RemoveContainer" containerID="64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4"
Jul 28 17:39:55.231215 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.231197    2578 scope.go:117] "RemoveContainer" containerID="8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b"
Jul 28 17:39:55.237945 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.237928    2578 scope.go:117] "RemoveContainer" containerID="fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67"
Jul 28 17:39:55.244447 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.244431    2578 scope.go:117] "RemoveContainer" containerID="64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4"
Jul 28 17:39:55.244711 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:39:55.244691    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4\": container with ID starting with 64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4 not found: ID does not exist" containerID="64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4"
Jul 28 17:39:55.244754 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.244722    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4"} err="failed to get container status \"64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4\": rpc error: code = NotFound desc = could not find container \"64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4\": container with ID starting with 64eff4d9792515afc229ea3e3c0530768414f1bb053fa9c2968a33e9f89d79f4 not found: ID does not exist"
Jul 28 17:39:55.244754 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.244740    2578 scope.go:117] "RemoveContainer" containerID="8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b"
Jul 28 17:39:55.244975 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:39:55.244958    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b\": container with ID starting with 8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b not found: ID does not exist" containerID="8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b"
Jul 28 17:39:55.245021 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.244979    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b"} err="failed to get container status \"8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b\": rpc error: code = NotFound desc = could not find container \"8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b\": container with ID starting with 8372c17c38ecd673211797074811d37fa92ece403891b3019196ee042533787b not found: ID does not exist"
Jul 28 17:39:55.245021 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.244993    2578 scope.go:117] "RemoveContainer" containerID="fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67"
Jul 28 17:39:55.245211 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:39:55.245186    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67\": container with ID starting with fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67 not found: ID does not exist" containerID="fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67"
Jul 28 17:39:55.245255 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.245213    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67"} err="failed to get container status \"fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67\": rpc error: code = NotFound desc = could not find container \"fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67\": container with ID starting with fa3b60b70a619d97d76737b35d5a1226f5e05e430d449ed3d11db9e46573bf67 not found: ID does not exist"
Jul 28 17:39:55.247418 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.247372    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"]
Jul 28 17:39:55.251965 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.251944    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-7w5kv"]
Jul 28 17:39:55.362266 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.362244    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" path="/var/lib/kubelet/pods/73464ed8-c6b6-426c-8ae9-c4d0822f30d4/volumes"
Jul 28 17:39:55.695853 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.695824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:55.778637 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.778607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:55.779170 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779146    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"]
Jul 28 17:39:55.779568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779552    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container"
Jul 28 17:39:55.779615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779572    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container"
Jul 28 17:39:55.779615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779590    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kube-rbac-proxy"
Jul 28 17:39:55.779615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779599    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kube-rbac-proxy"
Jul 28 17:39:55.779615 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779612    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="storage-initializer"
Jul 28 17:39:55.779739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="storage-initializer"
Jul 28 17:39:55.779739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779631    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container"
Jul 28 17:39:55.779739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779639    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container"
Jul 28 17:39:55.779739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779652    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kube-rbac-proxy"
Jul 28 17:39:55.779739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779659    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kube-rbac-proxy"
Jul 28 17:39:55.779739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779673    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="storage-initializer"
Jul 28 17:39:55.779739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779680    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="storage-initializer"
Jul 28 17:39:55.779951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779750    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kube-rbac-proxy"
Jul 28 17:39:55.779951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779760    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="73464ed8-c6b6-426c-8ae9-c4d0822f30d4" containerName="kserve-container"
Jul 28 17:39:55.779951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779770    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kube-rbac-proxy"
Jul 28 17:39:55.779951 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.779780    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3439b76-c6d0-49d4-960d-f3115bb96a70" containerName="kserve-container"
Jul 28 17:39:55.784577 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.784560    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"]
Jul 28 17:39:55.784695 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.784682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:55.787346 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 17:39:55.787731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:39:55.787731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:39:55.787731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787619    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:39:55.787731 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:55.787956 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w2njn\""
Jul 28 17:39:55.787956 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:39:55.787956 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.787814    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 17:39:55.790372 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.790351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 17:39:55.797153 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.797135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"]
Jul 28 17:39:55.798511 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.798493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"]
Jul 28 17:39:55.902339 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpxks\" (UniqueName: \"kubernetes.io/projected/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-api-access-zpxks\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xr6\" (UniqueName: \"kubernetes.io/projected/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-api-access-l8xr6\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:55.902669 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.902636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:55.931889 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:55.931867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:56.003791 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.003791 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xr6\" (UniqueName: \"kubernetes.io/projected/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-api-access-l8xr6\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.003791 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpxks\" (UniqueName: \"kubernetes.io/projected/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-api-access-zpxks\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:39:56.003956    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.003982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.004038 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:39:56.004035    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls podName:d66b6a0f-b3aa-40bc-ace9-897e56b625e5 nodeName:}" failed. No retries permitted until 2026-07-28 17:39:56.504012187 +0000 UTC m=+1751.669648013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls") pod "isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" (UID: "d66b6a0f-b3aa-40bc-ace9-897e56b625e5") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 17:39:56.004421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.004400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.004583 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.004556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.004683 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.004621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.004747 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.004707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.004747 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.004706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.004946 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.004928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.006716 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.006694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.013349 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.013328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xr6\" (UniqueName: \"kubernetes.io/projected/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-api-access-l8xr6\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.013477 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.013447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpxks\" (UniqueName: \"kubernetes.io/projected/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-api-access-zpxks\") pod \"isvc-canary-stop-predictor-5bf5667989-5jrxv\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.095848 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.095820    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:39:56.230930 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.230900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"]
Jul 28 17:39:56.233580 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:39:56.233557    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018f695b_7fe7_4e04_8197_b7eb8fe9b6fd.slice/crio-cf54b679732d88edba285e1339a491d5faac511ef71c98eb56c14af35d3568bf WatchSource:0}: Error finding container cf54b679732d88edba285e1339a491d5faac511ef71c98eb56c14af35d3568bf: Status 404 returned error can't find the container with id cf54b679732d88edba285e1339a491d5faac511ef71c98eb56c14af35d3568bf
Jul 28 17:39:56.341039 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.340998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:56.509592 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.509557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.511852 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.511834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.702525 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.702494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:39:56.833478 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:56.833432    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"]
Jul 28 17:39:56.835475 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:39:56.835448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b6a0f_b3aa_40bc_ace9_897e56b625e5.slice/crio-af1a8a4c691e8af756e1598a31c4b500d8a2c2922e344c8ca30966239b112bd3 WatchSource:0}: Error finding container af1a8a4c691e8af756e1598a31c4b500d8a2c2922e344c8ca30966239b112bd3: Status 404 returned error can't find the container with id af1a8a4c691e8af756e1598a31c4b500d8a2c2922e344c8ca30966239b112bd3
Jul 28 17:39:57.231739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:57.231703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerStarted","Data":"310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de"}
Jul 28 17:39:57.232225 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:57.231768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerStarted","Data":"af1a8a4c691e8af756e1598a31c4b500d8a2c2922e344c8ca30966239b112bd3"}
Jul 28 17:39:57.233360 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:57.233329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerStarted","Data":"3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0"}
Jul 28 17:39:57.233474 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:39:57.233404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerStarted","Data":"cf54b679732d88edba285e1339a491d5faac511ef71c98eb56c14af35d3568bf"}
Jul 28 17:40:00.251867 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:00.251776    2578 generic.go:356] "Generic (PLEG): container finished" podID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerID="3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0" exitCode=0
Jul 28 17:40:00.251867 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:00.251841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerDied","Data":"3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0"}
Jul 28 17:40:01.257056 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.256979    2578 generic.go:356] "Generic (PLEG): container finished" podID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerID="310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de" exitCode=0
Jul 28 17:40:01.257461 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.257055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerDied","Data":"310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de"}
Jul 28 17:40:01.259180 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.259158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerStarted","Data":"3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1"}
Jul 28 17:40:01.259256 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.259205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerStarted","Data":"b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5"}
Jul 28 17:40:01.259455 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.259436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:40:01.259531 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.259486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:40:01.260821 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.260799    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:01.305472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:01.305328    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podStartSLOduration=2.604501796 podStartE2EDuration="6.305309232s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.700807436s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:01.304865709 +0000 UTC m=+1756.470501545" watchObservedRunningTime="2026-07-28 17:40:01.305309232 +0000 UTC m=+1756.470945057"
Jul 28 17:40:02.265759 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:02.265728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerStarted","Data":"8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193"}
Jul 28 17:40:02.266196 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:02.265785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerStarted","Data":"4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157"}
Jul 28 17:40:02.266196 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:02.266155    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:02.266341 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:02.266231    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:40:02.266341 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:02.266335    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:40:02.267752 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:02.267730    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:40:02.289493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:02.289455    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podStartSLOduration=3.347963152 podStartE2EDuration="7.289443687s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941480535s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:02.287779844 +0000 UTC m=+1757.453415679" watchObservedRunningTime="2026-07-28 17:40:02.289443687 +0000 UTC m=+1757.455079521"
Jul 28 17:40:03.270419 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:03.270382    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:40:07.270347 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:07.270296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:40:07.270799 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:07.270773    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:08.274901 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:08.274875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:40:08.275598 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:08.275567    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:40:17.271313 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:17.271268    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:18.275798 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:18.275759    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:40:27.271225 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:27.271186    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:28.276198 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:28.276160    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:40:37.271371 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:37.271272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:38.275865 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:38.275828    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:40:47.271260 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:47.271214    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:48.275857 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:48.275815    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:40:57.271425 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:57.271385    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:40:58.276285 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:40:58.276247    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:41:07.270846 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:07.270808    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:41:08.275771 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:08.275731    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:41:08.358659 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:08.358619    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:41:17.365415 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:17.365381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:41:18.359018 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:18.358983    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:41:25.761267 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.761231    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"]
Jul 28 17:41:25.761905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.761525    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" containerID="cri-o://b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5" gracePeriod=30
Jul 28 17:41:25.761905 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.761558    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kube-rbac-proxy" containerID="cri-o://3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1" gracePeriod=30
Jul 28 17:41:25.805829 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.805796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:25.834945 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.834912    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"]
Jul 28 17:41:25.835188 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.835162    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" containerID="cri-o://4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157" gracePeriod=30
Jul 28 17:41:25.835329 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.835197    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kube-rbac-proxy" containerID="cri-o://8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193" gracePeriod=30
Jul 28 17:41:25.921198 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:25.921156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:26.592816 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:26.592787    2578 generic.go:356] "Generic (PLEG): container finished" podID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerID="3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1" exitCode=2
Jul 28 17:41:26.592994 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:26.592862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerDied","Data":"3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1"}
Jul 28 17:41:26.595154 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:26.595128    2578 generic.go:356] "Generic (PLEG): container finished" podID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerID="8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193" exitCode=2
Jul 28 17:41:26.595283 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:26.595159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerDied","Data":"8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193"}
Jul 28 17:41:27.266979 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:27.266932    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 17:41:27.359494 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:27.359452    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:41:28.270672 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:28.270631    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 17:41:28.358568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:28.358526    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:41:30.294295 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.294267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:41:30.362372 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.362347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpxks\" (UniqueName: \"kubernetes.io/projected/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-api-access-zpxks\") pod \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") "
Jul 28 17:41:30.362505 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.362430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kserve-provision-location\") pod \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") "
Jul 28 17:41:30.362505 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.362450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-cabundle-cert\") pod \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") "
Jul 28 17:41:30.362505 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.362475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-proxy-tls\") pod \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") "
Jul 28 17:41:30.362635 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.362507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-rbac-proxy-sar-config\") pod \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\" (UID: \"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd\") "
Jul 28 17:41:30.362782 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.362736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" (UID: "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:41:30.362782 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.362769    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" (UID: "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:30.363098 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.363004    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" (UID: "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:30.364590 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.364564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" (UID: "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:30.364678 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.364651    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-api-access-zpxks" (OuterVolumeSpecName: "kube-api-access-zpxks") pod "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" (UID: "018f695b-7fe7-4e04-8197-b7eb8fe9b6fd"). InnerVolumeSpecName "kube-api-access-zpxks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:30.463235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.463212    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpxks\" (UniqueName: \"kubernetes.io/projected/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-api-access-zpxks\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.463235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.463237    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.463413 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.463247    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.463413 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.463257    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.463413 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.463265    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.468428 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.468408    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:41:30.564238 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kserve-provision-location\") pod \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") "
Jul 28 17:41:30.564414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564248    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-cabundle-cert\") pod \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") "
Jul 28 17:41:30.564414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-rbac-proxy-sar-config\") pod \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") "
Jul 28 17:41:30.564414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8xr6\" (UniqueName: \"kubernetes.io/projected/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-api-access-l8xr6\") pod \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") "
Jul 28 17:41:30.564414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls\") pod \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\" (UID: \"d66b6a0f-b3aa-40bc-ace9-897e56b625e5\") "
Jul 28 17:41:30.564642 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d66b6a0f-b3aa-40bc-ace9-897e56b625e5" (UID: "d66b6a0f-b3aa-40bc-ace9-897e56b625e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:41:30.564701 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564643    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d66b6a0f-b3aa-40bc-ace9-897e56b625e5" (UID: "d66b6a0f-b3aa-40bc-ace9-897e56b625e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:30.564701 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.564647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d66b6a0f-b3aa-40bc-ace9-897e56b625e5" (UID: "d66b6a0f-b3aa-40bc-ace9-897e56b625e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:30.566327 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.566283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-api-access-l8xr6" (OuterVolumeSpecName: "kube-api-access-l8xr6") pod "d66b6a0f-b3aa-40bc-ace9-897e56b625e5" (UID: "d66b6a0f-b3aa-40bc-ace9-897e56b625e5"). InnerVolumeSpecName "kube-api-access-l8xr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:30.566426 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.566284    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d66b6a0f-b3aa-40bc-ace9-897e56b625e5" (UID: "d66b6a0f-b3aa-40bc-ace9-897e56b625e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:30.615243 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.615161    2578 generic.go:356] "Generic (PLEG): container finished" podID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerID="4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157" exitCode=0
Jul 28 17:41:30.615243 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.615231    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"
Jul 28 17:41:30.615463 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.615251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerDied","Data":"4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157"}
Jul 28 17:41:30.615463 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.615341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f" event={"ID":"d66b6a0f-b3aa-40bc-ace9-897e56b625e5","Type":"ContainerDied","Data":"af1a8a4c691e8af756e1598a31c4b500d8a2c2922e344c8ca30966239b112bd3"}
Jul 28 17:41:30.615463 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.615374    2578 scope.go:117] "RemoveContainer" containerID="8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193"
Jul 28 17:41:30.617193 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.617173    2578 generic.go:356] "Generic (PLEG): container finished" podID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerID="b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5" exitCode=0
Jul 28 17:41:30.617329 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.617222    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"
Jul 28 17:41:30.617329 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.617224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerDied","Data":"b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5"}
Jul 28 17:41:30.617462 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.617345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv" event={"ID":"018f695b-7fe7-4e04-8197-b7eb8fe9b6fd","Type":"ContainerDied","Data":"cf54b679732d88edba285e1339a491d5faac511ef71c98eb56c14af35d3568bf"}
Jul 28 17:41:30.623765 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.623746    2578 scope.go:117] "RemoveContainer" containerID="4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157"
Jul 28 17:41:30.631120 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.631100    2578 scope.go:117] "RemoveContainer" containerID="310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de"
Jul 28 17:41:30.637808 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.637793    2578 scope.go:117] "RemoveContainer" containerID="8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193"
Jul 28 17:41:30.638050 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:41:30.638032    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193\": container with ID starting with 8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193 not found: ID does not exist" containerID="8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193"
Jul 28 17:41:30.638100 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.638057    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193"} err="failed to get container status \"8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193\": rpc error: code = NotFound desc = could not find container \"8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193\": container with ID starting with 8e2743774b60276689c890a0c6260de74d9e258e815ca16db306ec09e934d193 not found: ID does not exist"
Jul 28 17:41:30.638100 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.638074    2578 scope.go:117] "RemoveContainer" containerID="4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157"
Jul 28 17:41:30.638325 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:41:30.638288    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157\": container with ID starting with 4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157 not found: ID does not exist" containerID="4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157"
Jul 28 17:41:30.638414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.638329    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157"} err="failed to get container status \"4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157\": rpc error: code = NotFound desc = could not find container \"4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157\": container with ID starting with 4b09358f0a300a85fbcb41104b35bfe92dd36a93a30c5e6df2e1af80f8e6a157 not found: ID does not exist"
Jul 28 17:41:30.638414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.638346    2578 scope.go:117] "RemoveContainer" containerID="310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de"
Jul 28 17:41:30.638568 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:41:30.638551    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de\": container with ID starting with 310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de not found: ID does not exist" containerID="310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de"
Jul 28 17:41:30.638618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.638573    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de"} err="failed to get container status \"310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de\": rpc error: code = NotFound desc = could not find container \"310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de\": container with ID starting with 310eb902d00dd143e1bd6a719461da05126d6c7e648ada90b175bb3300df38de not found: ID does not exist"
Jul 28 17:41:30.638618 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.638588    2578 scope.go:117] "RemoveContainer" containerID="3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1"
Jul 28 17:41:30.645739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.645721    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"]
Jul 28 17:41:30.645801 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.645771    2578 scope.go:117] "RemoveContainer" containerID="b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5"
Jul 28 17:41:30.649052 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.649028    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-q7b5f"]
Jul 28 17:41:30.653117 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.653093    2578 scope.go:117] "RemoveContainer" containerID="3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0"
Jul 28 17:41:30.659821 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.659796    2578 scope.go:117] "RemoveContainer" containerID="3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1"
Jul 28 17:41:30.660071 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:41:30.660043    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1\": container with ID starting with 3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1 not found: ID does not exist" containerID="3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1"
Jul 28 17:41:30.660170 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.660079    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1"} err="failed to get container status \"3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1\": rpc error: code = NotFound desc = could not find container \"3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1\": container with ID starting with 3764f4a6f2d70da11437f0ce4cb1a3e8fd09bd3b67fc200ede6452957a9602a1 not found: ID does not exist"
Jul 28 17:41:30.660170 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.660099    2578 scope.go:117] "RemoveContainer" containerID="b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5"
Jul 28 17:41:30.660388 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:41:30.660365    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5\": container with ID starting with b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5 not found: ID does not exist" containerID="b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5"
Jul 28 17:41:30.660467 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.660392    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5"} err="failed to get container status \"b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5\": rpc error: code = NotFound desc = could not find container \"b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5\": container with ID starting with b6af645b14253c09f2dda8de8e9c368c246eff89ae8954403548eba0c7f035f5 not found: ID does not exist"
Jul 28 17:41:30.660467 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.660406    2578 scope.go:117] "RemoveContainer" containerID="3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0"
Jul 28 17:41:30.660688 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:41:30.660648    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0\": container with ID starting with 3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0 not found: ID does not exist" containerID="3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0"
Jul 28 17:41:30.660763 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.660693    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0"} err="failed to get container status \"3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0\": rpc error: code = NotFound desc = could not find container \"3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0\": container with ID starting with 3ecc9cce64e252d218ebdc9666ee9d37ffcc437f779522f1cff8dd22b89591a0 not found: ID does not exist"
Jul 28 17:41:30.662530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.662511    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"]
Jul 28 17:41:30.665047 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.665030    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.665156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.665050    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.665156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.665065    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.665156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.665078    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.665156 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.665091    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8xr6\" (UniqueName: \"kubernetes.io/projected/d66b6a0f-b3aa-40bc-ace9-897e56b625e5-kube-api-access-l8xr6\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:41:30.666803 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:30.666782    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-5jrxv"]
Jul 28 17:41:31.362579 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:31.362506    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" path="/var/lib/kubelet/pods/018f695b-7fe7-4e04-8197-b7eb8fe9b6fd/volumes"
Jul 28 17:41:31.362991 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:31.362979    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" path="/var/lib/kubelet/pods/d66b6a0f-b3aa-40bc-ace9-897e56b625e5/volumes"
Jul 28 17:41:36.024855 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.024822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:36.073745 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.073706    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"]
Jul 28 17:41:36.074054 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074037    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="storage-initializer"
Jul 28 17:41:36.074126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074057    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="storage-initializer"
Jul 28 17:41:36.074126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074074    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container"
Jul 28 17:41:36.074126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074082    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container"
Jul 28 17:41:36.074126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074101    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kube-rbac-proxy"
Jul 28 17:41:36.074126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074110    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kube-rbac-proxy"
Jul 28 17:41:36.074126 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074123    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="storage-initializer"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074131    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="storage-initializer"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074148    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kube-rbac-proxy"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074156    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kube-rbac-proxy"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074170    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074178    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074243    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kube-rbac-proxy"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074257    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kube-rbac-proxy"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074266    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="018f695b-7fe7-4e04-8197-b7eb8fe9b6fd" containerName="kserve-container"
Jul 28 17:41:36.074454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.074276    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66b6a0f-b3aa-40bc-ace9-897e56b625e5" containerName="kserve-container"
Jul 28 17:41:36.079479 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.079458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.083012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.082987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w2njn\""
Jul 28 17:41:36.083143 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.083120    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b2ec6b-kube-rbac-proxy-sar-config\""
Jul 28 17:41:36.083230 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.083214    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:41:36.084455 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.084436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:41:36.084668 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.084650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b2ec6b-predictor-serving-cert\""
Jul 28 17:41:36.084832 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.084810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:41:36.084946 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.084853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:41:36.091456 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.091435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"]
Jul 28 17:41:36.104450 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.104423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.104556 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.104463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2924909b-b800-434d-a66a-df3e19fe8426-proxy-tls\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.104556 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.104490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtfb\" (UniqueName: \"kubernetes.io/projected/2924909b-b800-434d-a66a-df3e19fe8426-kube-api-access-vvtfb\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.104649 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.104606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2924909b-b800-434d-a66a-df3e19fe8426-kserve-provision-location\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.104649 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.104642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-cabundle-cert\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.192434 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.192401    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:36.205548 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.205521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2924909b-b800-434d-a66a-df3e19fe8426-kserve-provision-location\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.205671 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.205565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-cabundle-cert\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.205671 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.205600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.205671 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.205632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2924909b-b800-434d-a66a-df3e19fe8426-proxy-tls\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.205671 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.205656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtfb\" (UniqueName: \"kubernetes.io/projected/2924909b-b800-434d-a66a-df3e19fe8426-kube-api-access-vvtfb\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.205913 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.205895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2924909b-b800-434d-a66a-df3e19fe8426-kserve-provision-location\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.206272 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.206254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.206386 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.206288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-cabundle-cert\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.208080 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.208056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2924909b-b800-434d-a66a-df3e19fe8426-proxy-tls\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.216737 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.216718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtfb\" (UniqueName: \"kubernetes.io/projected/2924909b-b800-434d-a66a-df3e19fe8426-kube-api-access-vvtfb\") pod \"isvc-primary-b2ec6b-predictor-565df7c8b5-27fls\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") " pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.389760 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.389726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:36.521247 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.521217    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"]
Jul 28 17:41:36.524541 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:41:36.524512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2924909b_b800_434d_a66a_df3e19fe8426.slice/crio-62f1dde4075038e97fcec9a9f0caa3a8c218084791eb0c078b4ed23926fb97ac WatchSource:0}: Error finding container 62f1dde4075038e97fcec9a9f0caa3a8c218084791eb0c078b4ed23926fb97ac: Status 404 returned error can't find the container with id 62f1dde4075038e97fcec9a9f0caa3a8c218084791eb0c078b4ed23926fb97ac
Jul 28 17:41:36.646249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.646171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerStarted","Data":"c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1"}
Jul 28 17:41:36.646249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:36.646232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerStarted","Data":"62f1dde4075038e97fcec9a9f0caa3a8c218084791eb0c078b4ed23926fb97ac"}
Jul 28 17:41:40.661681 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:40.661599    2578 generic.go:356] "Generic (PLEG): container finished" podID="2924909b-b800-434d-a66a-df3e19fe8426" containerID="c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1" exitCode=0
Jul 28 17:41:40.661681 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:40.661639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerDied","Data":"c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1"}
Jul 28 17:41:41.667630 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:41.667597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerStarted","Data":"fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f"}
Jul 28 17:41:41.668012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:41.667662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerStarted","Data":"b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838"}
Jul 28 17:41:41.668012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:41.667951    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:41.697352 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:41.697284    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podStartSLOduration=1.896667877 podStartE2EDuration="5.697270165s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800602288s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:41.696882611 +0000 UTC m=+1856.862518446" watchObservedRunningTime="2026-07-28 17:41:41.697270165 +0000 UTC m=+1856.862906000"
Jul 28 17:41:42.671127 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:42.671092    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:42.672341 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:42.672318    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:41:43.674446 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:43.674407    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:41:48.680155 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:48.680123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:41:48.680612 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:48.680588    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:41:58.681279 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:41:58.681234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:08.681407 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:42:08.681322    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:18.681527 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:42:18.681489    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:28.680840 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:42:28.680800    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:38.680821 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:42:38.680777    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:48.680702 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:42:48.680663    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:55.367261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:42:55.367234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:43:11.351182 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.351147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:11.376235 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.376204    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"]
Jul 28 17:43:11.379524 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.379509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.382380 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.382362    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-b2ec6b\""
Jul 28 17:43:11.382488 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.382362    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b2ec6b-predictor-serving-cert\""
Jul 28 17:43:11.383106 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.383090    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-b2ec6b-dockercfg-2zdwp\""
Jul 28 17:43:11.383769 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.383754    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b2ec6b-kube-rbac-proxy-sar-config\""
Jul 28 17:43:11.391804 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.391783    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"]
Jul 28 17:43:11.478856 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.478825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.479006 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.478887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-cabundle-cert\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.479092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.479067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kserve-provision-location\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.479159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.479100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddc64e3a-8a9c-4169-9aab-390b55ddadca-proxy-tls\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.479200 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.479177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8f7\" (UniqueName: \"kubernetes.io/projected/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-api-access-fq8f7\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.494626 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.494584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:11.579986 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.579946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.580175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.579999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-cabundle-cert\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.580175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.580050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kserve-provision-location\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.580175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.580071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddc64e3a-8a9c-4169-9aab-390b55ddadca-proxy-tls\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.580175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.580106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8f7\" (UniqueName: \"kubernetes.io/projected/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-api-access-fq8f7\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.580587 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.580563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kserve-provision-location\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.580768 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.580746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.580829 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.580763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-cabundle-cert\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.582660 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.582643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddc64e3a-8a9c-4169-9aab-390b55ddadca-proxy-tls\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.589981 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.589956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8f7\" (UniqueName: \"kubernetes.io/projected/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-api-access-fq8f7\") pod \"isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") " pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.689634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.689551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:11.820056 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.819956    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"]
Jul 28 17:43:11.822665 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:43:11.822639    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc64e3a_8a9c_4169_9aab_390b55ddadca.slice/crio-c55698eb8a031778ebd8a1acfb646016ae0d8c8810cd0be55d7ef2bd3d977714 WatchSource:0}: Error finding container c55698eb8a031778ebd8a1acfb646016ae0d8c8810cd0be55d7ef2bd3d977714: Status 404 returned error can't find the container with id c55698eb8a031778ebd8a1acfb646016ae0d8c8810cd0be55d7ef2bd3d977714
Jul 28 17:43:11.988958 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.988867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct" event={"ID":"ddc64e3a-8a9c-4169-9aab-390b55ddadca","Type":"ContainerStarted","Data":"40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33"}
Jul 28 17:43:11.988958 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:11.988936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct" event={"ID":"ddc64e3a-8a9c-4169-9aab-390b55ddadca","Type":"ContainerStarted","Data":"c55698eb8a031778ebd8a1acfb646016ae0d8c8810cd0be55d7ef2bd3d977714"}
Jul 28 17:43:16.004909 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:16.004832    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_ddc64e3a-8a9c-4169-9aab-390b55ddadca/storage-initializer/0.log"
Jul 28 17:43:16.004909 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:16.004873    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerID="40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33" exitCode=1
Jul 28 17:43:16.005287 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:16.004953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct" event={"ID":"ddc64e3a-8a9c-4169-9aab-390b55ddadca","Type":"ContainerDied","Data":"40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33"}
Jul 28 17:43:17.010883 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:17.010855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_ddc64e3a-8a9c-4169-9aab-390b55ddadca/storage-initializer/0.log"
Jul 28 17:43:17.011241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:17.010925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct" event={"ID":"ddc64e3a-8a9c-4169-9aab-390b55ddadca","Type":"ContainerStarted","Data":"2cbe2851c0c3bdf18f37ca04397a67b1139afc6e14d07b1f2e69805aa181186f"}
Jul 28 17:43:22.030357 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:22.030263    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_ddc64e3a-8a9c-4169-9aab-390b55ddadca/storage-initializer/1.log"
Jul 28 17:43:22.030748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:22.030668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_ddc64e3a-8a9c-4169-9aab-390b55ddadca/storage-initializer/0.log"
Jul 28 17:43:22.030748 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:22.030699    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerID="2cbe2851c0c3bdf18f37ca04397a67b1139afc6e14d07b1f2e69805aa181186f" exitCode=1
Jul 28 17:43:22.030820 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:22.030760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct" event={"ID":"ddc64e3a-8a9c-4169-9aab-390b55ddadca","Type":"ContainerDied","Data":"2cbe2851c0c3bdf18f37ca04397a67b1139afc6e14d07b1f2e69805aa181186f"}
Jul 28 17:43:22.030820 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:22.030814    2578 scope.go:117] "RemoveContainer" containerID="40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33"
Jul 28 17:43:22.031184 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:22.031156    2578 scope.go:117] "RemoveContainer" containerID="40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33"
Jul 28 17:43:22.041182 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:43:22.041162    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_kserve-ci-e2e-test(ddc64e3a-8a9c-4169-9aab-390b55ddadca)\"" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca"
Jul 28 17:43:22.041430 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:43:22.041403    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_kserve-ci-e2e-test_ddc64e3a-8a9c-4169-9aab-390b55ddadca_0 in pod sandbox c55698eb8a031778ebd8a1acfb646016ae0d8c8810cd0be55d7ef2bd3d977714 from index: no such id: '40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33'" containerID="40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33"
Jul 28 17:43:22.041486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:22.041442    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_kserve-ci-e2e-test_ddc64e3a-8a9c-4169-9aab-390b55ddadca_0 in pod sandbox c55698eb8a031778ebd8a1acfb646016ae0d8c8810cd0be55d7ef2bd3d977714 from index: no such id: '40d3855c8a0e095cda0ffd4cb8fe2cea5166368e26b78927e3c38ec662301a33'"
Jul 28 17:43:23.036866 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:23.036839    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_ddc64e3a-8a9c-4169-9aab-390b55ddadca/storage-initializer/1.log"
Jul 28 17:43:27.372852 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.372820    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"]
Jul 28 17:43:27.373840 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.373798    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container" containerID="cri-o://b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838" gracePeriod=30
Jul 28 17:43:27.374758 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.374113    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kube-rbac-proxy" containerID="cri-o://fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f" gracePeriod=30
Jul 28 17:43:27.434041 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.434007    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:27.440491 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.440464    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"]
Jul 28 17:43:27.561889 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.561865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_ddc64e3a-8a9c-4169-9aab-390b55ddadca/storage-initializer/1.log"
Jul 28 17:43:27.564175 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.564155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:27.716755 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.716668    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-rbac-proxy-sar-config\") pod \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") "
Jul 28 17:43:27.716755 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.716700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-cabundle-cert\") pod \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") "
Jul 28 17:43:27.716992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.716892    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kserve-provision-location\") pod \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") "
Jul 28 17:43:27.716992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.716941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddc64e3a-8a9c-4169-9aab-390b55ddadca-proxy-tls\") pod \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") "
Jul 28 17:43:27.716992 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.716981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq8f7\" (UniqueName: \"kubernetes.io/projected/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-api-access-fq8f7\") pod \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\" (UID: \"ddc64e3a-8a9c-4169-9aab-390b55ddadca\") "
Jul 28 17:43:27.717162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.717032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ddc64e3a-8a9c-4169-9aab-390b55ddadca" (UID: "ddc64e3a-8a9c-4169-9aab-390b55ddadca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:27.717162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.717059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ddc64e3a-8a9c-4169-9aab-390b55ddadca" (UID: "ddc64e3a-8a9c-4169-9aab-390b55ddadca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:27.717162 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.717145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddc64e3a-8a9c-4169-9aab-390b55ddadca" (UID: "ddc64e3a-8a9c-4169-9aab-390b55ddadca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:27.717270 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.717228    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:27.717270 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.717247    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:27.717270 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.717261    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddc64e3a-8a9c-4169-9aab-390b55ddadca-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:27.718939 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.718906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc64e3a-8a9c-4169-9aab-390b55ddadca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddc64e3a-8a9c-4169-9aab-390b55ddadca" (UID: "ddc64e3a-8a9c-4169-9aab-390b55ddadca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:27.719056 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.718980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-api-access-fq8f7" (OuterVolumeSpecName: "kube-api-access-fq8f7") pod "ddc64e3a-8a9c-4169-9aab-390b55ddadca" (UID: "ddc64e3a-8a9c-4169-9aab-390b55ddadca"). InnerVolumeSpecName "kube-api-access-fq8f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:27.818190 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.818151    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddc64e3a-8a9c-4169-9aab-390b55ddadca-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:27.818190 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:27.818183    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8f7\" (UniqueName: \"kubernetes.io/projected/ddc64e3a-8a9c-4169-9aab-390b55ddadca-kube-api-access-fq8f7\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:28.055730 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.055661    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct_ddc64e3a-8a9c-4169-9aab-390b55ddadca/storage-initializer/1.log"
Jul 28 17:43:28.055876 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.055760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"
Jul 28 17:43:28.055876 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.055785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct" event={"ID":"ddc64e3a-8a9c-4169-9aab-390b55ddadca","Type":"ContainerDied","Data":"c55698eb8a031778ebd8a1acfb646016ae0d8c8810cd0be55d7ef2bd3d977714"}
Jul 28 17:43:28.055876 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.055854    2578 scope.go:117] "RemoveContainer" containerID="2cbe2851c0c3bdf18f37ca04397a67b1139afc6e14d07b1f2e69805aa181186f"
Jul 28 17:43:28.058201 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.058180    2578 generic.go:356] "Generic (PLEG): container finished" podID="2924909b-b800-434d-a66a-df3e19fe8426" containerID="fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f" exitCode=2
Jul 28 17:43:28.058327 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.058232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerDied","Data":"fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f"}
Jul 28 17:43:28.102818 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.102790    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"]
Jul 28 17:43:28.104848 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.104813    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b2ec6b-predictor-6fcc99f698-jg9ct"]
Jul 28 17:43:28.674708 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.674667    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 17:43:28.714537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.714503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:28.884133 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:28.884102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:29.363191 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:29.363159    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" path="/var/lib/kubelet/pods/ddc64e3a-8a9c-4169-9aab-390b55ddadca/volumes"
Jul 28 17:43:31.717103 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.717076    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:43:31.853560 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.853529    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-cabundle-cert\") pod \"2924909b-b800-434d-a66a-df3e19fe8426\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") "
Jul 28 17:43:31.853735 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.853613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-kube-rbac-proxy-sar-config\") pod \"2924909b-b800-434d-a66a-df3e19fe8426\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") "
Jul 28 17:43:31.853735 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.853645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2924909b-b800-434d-a66a-df3e19fe8426-proxy-tls\") pod \"2924909b-b800-434d-a66a-df3e19fe8426\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") "
Jul 28 17:43:31.853735 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.853681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2924909b-b800-434d-a66a-df3e19fe8426-kserve-provision-location\") pod \"2924909b-b800-434d-a66a-df3e19fe8426\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") "
Jul 28 17:43:31.853893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.853737    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvtfb\" (UniqueName: \"kubernetes.io/projected/2924909b-b800-434d-a66a-df3e19fe8426-kube-api-access-vvtfb\") pod \"2924909b-b800-434d-a66a-df3e19fe8426\" (UID: \"2924909b-b800-434d-a66a-df3e19fe8426\") "
Jul 28 17:43:31.854038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.853915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2924909b-b800-434d-a66a-df3e19fe8426" (UID: "2924909b-b800-434d-a66a-df3e19fe8426"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:31.854038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.854024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2924909b-b800-434d-a66a-df3e19fe8426" (UID: "2924909b-b800-434d-a66a-df3e19fe8426"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:31.854198 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.854026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2924909b-b800-434d-a66a-df3e19fe8426-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2924909b-b800-434d-a66a-df3e19fe8426" (UID: "2924909b-b800-434d-a66a-df3e19fe8426"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:31.855738 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.855721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2924909b-b800-434d-a66a-df3e19fe8426-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2924909b-b800-434d-a66a-df3e19fe8426" (UID: "2924909b-b800-434d-a66a-df3e19fe8426"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:31.855839 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.855823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2924909b-b800-434d-a66a-df3e19fe8426-kube-api-access-vvtfb" (OuterVolumeSpecName: "kube-api-access-vvtfb") pod "2924909b-b800-434d-a66a-df3e19fe8426" (UID: "2924909b-b800-434d-a66a-df3e19fe8426"). InnerVolumeSpecName "kube-api-access-vvtfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:31.954532 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.954504    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:31.954532 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.954527    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2924909b-b800-434d-a66a-df3e19fe8426-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:31.954532 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.954538    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2924909b-b800-434d-a66a-df3e19fe8426-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:31.954740 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.954547    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtfb\" (UniqueName: \"kubernetes.io/projected/2924909b-b800-434d-a66a-df3e19fe8426-kube-api-access-vvtfb\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:31.954740 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:31.954556    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2924909b-b800-434d-a66a-df3e19fe8426-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:32.081073 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.081039    2578 generic.go:356] "Generic (PLEG): container finished" podID="2924909b-b800-434d-a66a-df3e19fe8426" containerID="b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838" exitCode=0
Jul 28 17:43:32.081255 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.081096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerDied","Data":"b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838"}
Jul 28 17:43:32.081255 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.081107    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"
Jul 28 17:43:32.081255 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.081183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls" event={"ID":"2924909b-b800-434d-a66a-df3e19fe8426","Type":"ContainerDied","Data":"62f1dde4075038e97fcec9a9f0caa3a8c218084791eb0c078b4ed23926fb97ac"}
Jul 28 17:43:32.081255 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.081221    2578 scope.go:117] "RemoveContainer" containerID="fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f"
Jul 28 17:43:32.090154 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.090139    2578 scope.go:117] "RemoveContainer" containerID="b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838"
Jul 28 17:43:32.098995 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.098978    2578 scope.go:117] "RemoveContainer" containerID="c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1"
Jul 28 17:43:32.105800 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.105777    2578 scope.go:117] "RemoveContainer" containerID="fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f"
Jul 28 17:43:32.106028 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:43:32.106001    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f\": container with ID starting with fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f not found: ID does not exist" containerID="fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f"
Jul 28 17:43:32.106121 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.106036    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f"} err="failed to get container status \"fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f\": rpc error: code = NotFound desc = could not find container \"fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f\": container with ID starting with fe3df5df81c0851150f89f5c897a607256a3a985c699b9b4dddb19135607ea5f not found: ID does not exist"
Jul 28 17:43:32.106121 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.106061    2578 scope.go:117] "RemoveContainer" containerID="b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838"
Jul 28 17:43:32.106336 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:43:32.106316    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838\": container with ID starting with b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838 not found: ID does not exist" containerID="b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838"
Jul 28 17:43:32.106421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.106346    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838"} err="failed to get container status \"b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838\": rpc error: code = NotFound desc = could not find container \"b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838\": container with ID starting with b525fb38218315f8c4bc9da51727dd6ae5de761451dd86b5dcf33f4aba506838 not found: ID does not exist"
Jul 28 17:43:32.106421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.106369    2578 scope.go:117] "RemoveContainer" containerID="c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1"
Jul 28 17:43:32.106421 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.106394    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"]
Jul 28 17:43:32.106600 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:43:32.106581    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1\": container with ID starting with c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1 not found: ID does not exist" containerID="c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1"
Jul 28 17:43:32.106638 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.106605    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1"} err="failed to get container status \"c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1\": rpc error: code = NotFound desc = could not find container \"c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1\": container with ID starting with c46ed3d6461d001d10d16a40db4600489727d691cf061f9082e144a4273041b1 not found: ID does not exist"
Jul 28 17:43:32.111087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.111067    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b2ec6b-predictor-565df7c8b5-27fls"]
Jul 28 17:43:32.611147 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.611110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:32.629295 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629263    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"]
Jul 28 17:43:32.629613 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="storage-initializer"
Jul 28 17:43:32.629710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629616    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="storage-initializer"
Jul 28 17:43:32.629710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629631    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerName="storage-initializer"
Jul 28 17:43:32.629710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629639    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerName="storage-initializer"
Jul 28 17:43:32.629710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629662    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container"
Jul 28 17:43:32.629710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629671    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container"
Jul 28 17:43:32.629710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629682    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kube-rbac-proxy"
Jul 28 17:43:32.629710 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629691    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kube-rbac-proxy"
Jul 28 17:43:32.630053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629763    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kube-rbac-proxy"
Jul 28 17:43:32.630053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629777    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerName="storage-initializer"
Jul 28 17:43:32.630053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629789    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2924909b-b800-434d-a66a-df3e19fe8426" containerName="kserve-container"
Jul 28 17:43:32.630053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629870    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerName="storage-initializer"
Jul 28 17:43:32.630053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629879    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerName="storage-initializer"
Jul 28 17:43:32.630053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.629940    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc64e3a-8a9c-4169-9aab-390b55ddadca" containerName="storage-initializer"
Jul 28 17:43:32.634315 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.634282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.637482 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.637463    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6e1105-predictor-serving-cert\""
Jul 28 17:43:32.637587 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.637504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-6e1105\""
Jul 28 17:43:32.637587 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.637505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:43:32.637978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.637957    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:43:32.637978 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.637971    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:43:32.638115 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.638017    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6e1105-kube-rbac-proxy-sar-config\""
Jul 28 17:43:32.638394 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.638379    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-6e1105-dockercfg-8qmmf\""
Jul 28 17:43:32.645915 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.645893    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"]
Jul 28 17:43:32.760582 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.760549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5c09df9-5986-4bb5-8b6e-65b1bca64046-proxy-tls\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.760939 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.760586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njrfl\" (UniqueName: \"kubernetes.io/projected/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-api-access-njrfl\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.760939 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.760615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kserve-provision-location\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.760939 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.760667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-cabundle-cert\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.760939 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.760708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.782685 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.782649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:32.861895 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.861802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5c09df9-5986-4bb5-8b6e-65b1bca64046-proxy-tls\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.861895 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.861858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njrfl\" (UniqueName: \"kubernetes.io/projected/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-api-access-njrfl\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.861895 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.861890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kserve-provision-location\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.862159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.861910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-cabundle-cert\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.862159 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.861933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.862393 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.862366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kserve-provision-location\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.862636 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.862613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-cabundle-cert\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.862696 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.862678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.864265 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.864238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5c09df9-5986-4bb5-8b6e-65b1bca64046-proxy-tls\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.877271 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.877251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njrfl\" (UniqueName: \"kubernetes.io/projected/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-api-access-njrfl\") pod \"isvc-init-fail-6e1105-predictor-9687fd768-jn8gv\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") " pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:32.944261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:32.944228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:33.073519 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:33.073485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"]
Jul 28 17:43:33.076352 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:43:33.076321    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c09df9_5986_4bb5_8b6e_65b1bca64046.slice/crio-c90a8917d7f30c5b600cbcb7bd234893da62adb5398212e5bb575cb2642b3771 WatchSource:0}: Error finding container c90a8917d7f30c5b600cbcb7bd234893da62adb5398212e5bb575cb2642b3771: Status 404 returned error can't find the container with id c90a8917d7f30c5b600cbcb7bd234893da62adb5398212e5bb575cb2642b3771
Jul 28 17:43:33.087167 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:33.087145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv" event={"ID":"f5c09df9-5986-4bb5-8b6e-65b1bca64046","Type":"ContainerStarted","Data":"c90a8917d7f30c5b600cbcb7bd234893da62adb5398212e5bb575cb2642b3771"}
Jul 28 17:43:33.363909 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:33.363831    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2924909b-b800-434d-a66a-df3e19fe8426" path="/var/lib/kubelet/pods/2924909b-b800-434d-a66a-df3e19fe8426/volumes"
Jul 28 17:43:34.092308 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:34.092263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv" event={"ID":"f5c09df9-5986-4bb5-8b6e-65b1bca64046","Type":"ContainerStarted","Data":"1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a"}
Jul 28 17:43:41.120081 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:41.120051    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6e1105-predictor-9687fd768-jn8gv_f5c09df9-5986-4bb5-8b6e-65b1bca64046/storage-initializer/0.log"
Jul 28 17:43:41.120476 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:41.120086    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerID="1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a" exitCode=1
Jul 28 17:43:41.120476 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:41.120163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv" event={"ID":"f5c09df9-5986-4bb5-8b6e-65b1bca64046","Type":"ContainerDied","Data":"1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a"}
Jul 28 17:43:42.125719 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:42.125688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6e1105-predictor-9687fd768-jn8gv_f5c09df9-5986-4bb5-8b6e-65b1bca64046/storage-initializer/0.log"
Jul 28 17:43:42.126249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:42.125812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv" event={"ID":"f5c09df9-5986-4bb5-8b6e-65b1bca64046","Type":"ContainerStarted","Data":"9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d"}
Jul 28 17:43:42.660003 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:42.659969    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"]
Jul 28 17:43:42.717986 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:42.717943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:43.129465 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:43.129423    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerName="storage-initializer" containerID="cri-o://9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d" gracePeriod=30
Jul 28 17:43:47.578376 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.578354    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6e1105-predictor-9687fd768-jn8gv_f5c09df9-5986-4bb5-8b6e-65b1bca64046/storage-initializer/1.log"
Jul 28 17:43:47.578777 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.578760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6e1105-predictor-9687fd768-jn8gv_f5c09df9-5986-4bb5-8b6e-65b1bca64046/storage-initializer/0.log"
Jul 28 17:43:47.581149 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.581136    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:47.687392 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kserve-provision-location\") pod \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") "
Jul 28 17:43:47.687392 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njrfl\" (UniqueName: \"kubernetes.io/projected/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-api-access-njrfl\") pod \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") "
Jul 28 17:43:47.687624 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5c09df9-5986-4bb5-8b6e-65b1bca64046-proxy-tls\") pod \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") "
Jul 28 17:43:47.687624 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-rbac-proxy-sar-config\") pod \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") "
Jul 28 17:43:47.687624 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687506    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-cabundle-cert\") pod \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\" (UID: \"f5c09df9-5986-4bb5-8b6e-65b1bca64046\") "
Jul 28 17:43:47.687624 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5c09df9-5986-4bb5-8b6e-65b1bca64046" (UID: "f5c09df9-5986-4bb5-8b6e-65b1bca64046"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:47.687837 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687747    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:47.687884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5c09df9-5986-4bb5-8b6e-65b1bca64046" (UID: "f5c09df9-5986-4bb5-8b6e-65b1bca64046"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:47.687884 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.687843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5c09df9-5986-4bb5-8b6e-65b1bca64046" (UID: "f5c09df9-5986-4bb5-8b6e-65b1bca64046"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:47.689401 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.689380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-api-access-njrfl" (OuterVolumeSpecName: "kube-api-access-njrfl") pod "f5c09df9-5986-4bb5-8b6e-65b1bca64046" (UID: "f5c09df9-5986-4bb5-8b6e-65b1bca64046"). InnerVolumeSpecName "kube-api-access-njrfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:47.689488 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.689466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09df9-5986-4bb5-8b6e-65b1bca64046-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5c09df9-5986-4bb5-8b6e-65b1bca64046" (UID: "f5c09df9-5986-4bb5-8b6e-65b1bca64046"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:47.788278 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.788245    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njrfl\" (UniqueName: \"kubernetes.io/projected/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-api-access-njrfl\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:47.788278 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.788275    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5c09df9-5986-4bb5-8b6e-65b1bca64046-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:47.788506 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.788291    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:47.788506 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.788317    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5c09df9-5986-4bb5-8b6e-65b1bca64046-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:43:47.856072 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.856037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:47.887893 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.887864    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"]
Jul 28 17:43:47.888142 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.888130    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerName="storage-initializer"
Jul 28 17:43:47.888200 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.888143    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerName="storage-initializer"
Jul 28 17:43:47.888200 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.888198    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerName="storage-initializer"
Jul 28 17:43:47.888277 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.888207    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerName="storage-initializer"
Jul 28 17:43:47.888277 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.888260    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerName="storage-initializer"
Jul 28 17:43:47.888277 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.888266    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerName="storage-initializer"
Jul 28 17:43:47.891470 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.891452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:47.894394 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.894376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:43:47.894635 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.894621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-31643-predictor-serving-cert\""
Jul 28 17:43:47.894758 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.894743    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w2njn\""
Jul 28 17:43:47.894891 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.894878    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-31643-kube-rbac-proxy-sar-config\""
Jul 28 17:43:47.904923 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.904904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"]
Jul 28 17:43:47.990140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.990061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:47.990140 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.990099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kserve-provision-location\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:47.990320 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.990171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-proxy-tls\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:47.990320 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.990218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlt2m\" (UniqueName: \"kubernetes.io/projected/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-api-access-mlt2m\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:47.990320 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:47.990236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-cabundle-cert\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.090824 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.090796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.090957 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.090829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kserve-provision-location\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.090957 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.090865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-proxy-tls\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.090957 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.090896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlt2m\" (UniqueName: \"kubernetes.io/projected/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-api-access-mlt2m\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.090957 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.090913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-cabundle-cert\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.091268 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.091251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kserve-provision-location\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.091603 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.091583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.091656 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.091611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-cabundle-cert\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.093224 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.093207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-proxy-tls\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.102746 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.102718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlt2m\" (UniqueName: \"kubernetes.io/projected/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-api-access-mlt2m\") pod \"raw-sklearn-31643-predictor-678589b5c6-64zrl\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") " pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.147237 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.147207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:48.149729 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.149711    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6e1105-predictor-9687fd768-jn8gv_f5c09df9-5986-4bb5-8b6e-65b1bca64046/storage-initializer/1.log"
Jul 28 17:43:48.150079 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.150065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6e1105-predictor-9687fd768-jn8gv_f5c09df9-5986-4bb5-8b6e-65b1bca64046/storage-initializer/0.log"
Jul 28 17:43:48.150125 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.150102    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" containerID="9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d" exitCode=1
Jul 28 17:43:48.150158 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.150148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv" event={"ID":"f5c09df9-5986-4bb5-8b6e-65b1bca64046","Type":"ContainerDied","Data":"9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d"}
Jul 28 17:43:48.150194 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.150163    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"
Jul 28 17:43:48.150225 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.150207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv" event={"ID":"f5c09df9-5986-4bb5-8b6e-65b1bca64046","Type":"ContainerDied","Data":"c90a8917d7f30c5b600cbcb7bd234893da62adb5398212e5bb575cb2642b3771"}
Jul 28 17:43:48.150259 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.150244    2578 scope.go:117] "RemoveContainer" containerID="9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d"
Jul 28 17:43:48.159171 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.159157    2578 scope.go:117] "RemoveContainer" containerID="1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a"
Jul 28 17:43:48.166200 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.166182    2578 scope.go:117] "RemoveContainer" containerID="9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d"
Jul 28 17:43:48.166454 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:43:48.166437    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d\": container with ID starting with 9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d not found: ID does not exist" containerID="9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d"
Jul 28 17:43:48.166507 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.166463    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d"} err="failed to get container status \"9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d\": rpc error: code = NotFound desc = could not find container \"9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d\": container with ID starting with 9676f83d46b30aa9961b772a7ab255cf27158c81d6939a4192bbc1507383451d not found: ID does not exist"
Jul 28 17:43:48.166507 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.166478    2578 scope.go:117] "RemoveContainer" containerID="1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a"
Jul 28 17:43:48.166690 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:43:48.166676    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a\": container with ID starting with 1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a not found: ID does not exist" containerID="1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a"
Jul 28 17:43:48.166736 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.166694    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a"} err="failed to get container status \"1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a\": rpc error: code = NotFound desc = could not find container \"1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a\": container with ID starting with 1c2da01567c8f487e8f28be3bb1bcf5938411dd8a5f4cf97e74e7580035b305a not found: ID does not exist"
Jul 28 17:43:48.193141 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.193119    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"]
Jul 28 17:43:48.197055 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.197034    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6e1105-predictor-9687fd768-jn8gv"]
Jul 28 17:43:48.201161 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.201141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:48.329087 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:48.329005    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"]
Jul 28 17:43:48.331551 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:43:48.331527    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2db35c_3d4b_4218_aee0_e38bcd5b3248.slice/crio-7ed1185f6da96b74722a44202de1017dfdaf0d623e9fa82c310f817a2382fc0a WatchSource:0}: Error finding container 7ed1185f6da96b74722a44202de1017dfdaf0d623e9fa82c310f817a2382fc0a: Status 404 returned error can't find the container with id 7ed1185f6da96b74722a44202de1017dfdaf0d623e9fa82c310f817a2382fc0a
Jul 28 17:43:49.155727 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:49.155683    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerStarted","Data":"10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4"}
Jul 28 17:43:49.156176 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:49.155759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerStarted","Data":"7ed1185f6da96b74722a44202de1017dfdaf0d623e9fa82c310f817a2382fc0a"}
Jul 28 17:43:49.362702 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:49.362672    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c09df9-5986-4bb5-8b6e-65b1bca64046" path="/var/lib/kubelet/pods/f5c09df9-5986-4bb5-8b6e-65b1bca64046/volumes"
Jul 28 17:43:53.172975 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:53.172939    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerID="10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4" exitCode=0
Jul 28 17:43:53.173390 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:53.172995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerDied","Data":"10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4"}
Jul 28 17:43:54.178830 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:54.178795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerStarted","Data":"55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303"}
Jul 28 17:43:54.179224 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:54.178853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerStarted","Data":"4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9"}
Jul 28 17:43:54.179224 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:54.179060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:54.204249 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:54.204204    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podStartSLOduration=3.310952326 podStartE2EDuration="7.204189481s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.893237155s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:54.202699474 +0000 UTC m=+1989.368335309" watchObservedRunningTime="2026-07-28 17:43:54.204189481 +0000 UTC m=+1989.369825314"
Jul 28 17:43:55.183539 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:55.183512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:43:55.184687 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:55.184661    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:43:56.187574 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:43:56.187538    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:44:01.193036 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:01.193005    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:44:01.193504 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:01.193483    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:44:11.194185 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:11.194144    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:44:20.824383 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:20.824341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:20.925129 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:20.925097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:21.194237 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:21.194138    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:44:31.194074 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:31.194033    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:44:41.194165 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:41.194126    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:44:51.194062 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:44:51.194023    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:45:01.193933 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:01.193883    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:45:07.362073 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:07.362004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:45:08.105921 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:08.105887    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"]
Jul 28 17:45:08.128523 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:08.128488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:08.458813 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:08.455441    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container" containerID="cri-o://4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9" gracePeriod=30
Jul 28 17:45:08.458813 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:08.455730    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kube-rbac-proxy" containerID="cri-o://55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303" gracePeriod=30
Jul 28 17:45:09.460151 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:09.460118    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerID="55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303" exitCode=2
Jul 28 17:45:09.460537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:09.460173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerDied","Data":"55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303"}
Jul 28 17:45:11.188084 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:11.188040    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 17:45:13.007443 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.007418    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:45:13.053568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.053505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-proxy-tls\") pod \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") "
Jul 28 17:45:13.053568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.053552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlt2m\" (UniqueName: \"kubernetes.io/projected/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-api-access-mlt2m\") pod \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") "
Jul 28 17:45:13.053756 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.053691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kserve-provision-location\") pod \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") "
Jul 28 17:45:13.053756 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.053740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-cabundle-cert\") pod \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") "
Jul 28 17:45:13.053867 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.053780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-rbac-proxy-sar-config\") pod \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\" (UID: \"0d2db35c-3d4b-4218-aee0-e38bcd5b3248\") "
Jul 28 17:45:13.054038 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.054004    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d2db35c-3d4b-4218-aee0-e38bcd5b3248" (UID: "0d2db35c-3d4b-4218-aee0-e38bcd5b3248"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:13.054148 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.054109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d2db35c-3d4b-4218-aee0-e38bcd5b3248" (UID: "0d2db35c-3d4b-4218-aee0-e38bcd5b3248"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:13.054244 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.054216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d2db35c-3d4b-4218-aee0-e38bcd5b3248" (UID: "0d2db35c-3d4b-4218-aee0-e38bcd5b3248"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:13.055634 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.055610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-api-access-mlt2m" (OuterVolumeSpecName: "kube-api-access-mlt2m") pod "0d2db35c-3d4b-4218-aee0-e38bcd5b3248" (UID: "0d2db35c-3d4b-4218-aee0-e38bcd5b3248"). InnerVolumeSpecName "kube-api-access-mlt2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:13.055696 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.055637    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d2db35c-3d4b-4218-aee0-e38bcd5b3248" (UID: "0d2db35c-3d4b-4218-aee0-e38bcd5b3248"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:13.116841 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.116808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:13.153856 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.153828    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"]
Jul 28 17:45:13.154105 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154093    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="storage-initializer"
Jul 28 17:45:13.154145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154107    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="storage-initializer"
Jul 28 17:45:13.154145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154117    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container"
Jul 28 17:45:13.154145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154122    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container"
Jul 28 17:45:13.154145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154134    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kube-rbac-proxy"
Jul 28 17:45:13.154145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154140    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kube-rbac-proxy"
Jul 28 17:45:13.154294 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154186    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kserve-container"
Jul 28 17:45:13.154294 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154194    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerName="kube-rbac-proxy"
Jul 28 17:45:13.154800 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154780    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlt2m\" (UniqueName: \"kubernetes.io/projected/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-api-access-mlt2m\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:45:13.154882 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154803    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:45:13.154882 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154817    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:45:13.154882 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154833    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:45:13.154882 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.154845    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d2db35c-3d4b-4218-aee0-e38bcd5b3248-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:45:13.157375 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.157360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.160454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.160434    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-81166-kube-rbac-proxy-sar-config\""
Jul 28 17:45:13.160454 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.160454    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-81166-predictor-serving-cert\""
Jul 28 17:45:13.175622 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.175603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"]
Jul 28 17:45:13.200321 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.200268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:13.255949 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.255926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kserve-provision-location\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.256051 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.255963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-cabundle-cert\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.256051 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.255985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-proxy-tls\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.256146 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.256051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.256146 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.256118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhzc\" (UniqueName: \"kubernetes.io/projected/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-api-access-9bhzc\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357032 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhzc\" (UniqueName: \"kubernetes.io/projected/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-api-access-9bhzc\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kserve-provision-location\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357145 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-cabundle-cert\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357240 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-proxy-tls\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357290 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357520 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kserve-provision-location\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357782 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-cabundle-cert\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.357890 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.357859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.359705 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.359679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-proxy-tls\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.366363 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.366339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhzc\" (UniqueName: \"kubernetes.io/projected/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-api-access-9bhzc\") pod \"raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.467604 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.467580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:13.476065 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.476036    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" containerID="4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9" exitCode=0
Jul 28 17:45:13.476160 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.476082    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"
Jul 28 17:45:13.476160 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.476125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerDied","Data":"4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9"}
Jul 28 17:45:13.476291 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.476197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl" event={"ID":"0d2db35c-3d4b-4218-aee0-e38bcd5b3248","Type":"ContainerDied","Data":"7ed1185f6da96b74722a44202de1017dfdaf0d623e9fa82c310f817a2382fc0a"}
Jul 28 17:45:13.476291 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.476239    2578 scope.go:117] "RemoveContainer" containerID="55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303"
Jul 28 17:45:13.486219 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.486185    2578 scope.go:117] "RemoveContainer" containerID="4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9"
Jul 28 17:45:13.493937 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.493920    2578 scope.go:117] "RemoveContainer" containerID="10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4"
Jul 28 17:45:13.498165 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.498144    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"]
Jul 28 17:45:13.501996 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.501975    2578 scope.go:117] "RemoveContainer" containerID="55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303"
Jul 28 17:45:13.502358 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:45:13.502290    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303\": container with ID starting with 55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303 not found: ID does not exist" containerID="55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303"
Jul 28 17:45:13.502358 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.502340    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303"} err="failed to get container status \"55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303\": rpc error: code = NotFound desc = could not find container \"55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303\": container with ID starting with 55273016b500ae63d4d256ecc2f4c08847ea74dd8a72cde493d845047c71e303 not found: ID does not exist"
Jul 28 17:45:13.502537 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.502363    2578 scope.go:117] "RemoveContainer" containerID="4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9"
Jul 28 17:45:13.502613 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:45:13.502595    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9\": container with ID starting with 4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9 not found: ID does not exist" containerID="4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9"
Jul 28 17:45:13.502670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.502605    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-31643-predictor-678589b5c6-64zrl"]
Jul 28 17:45:13.502670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.502615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9"} err="failed to get container status \"4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9\": rpc error: code = NotFound desc = could not find container \"4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9\": container with ID starting with 4bcb7e26e04d0abb6aac58d255458ab2b6509732e58ae18d9d9e52b0ee0719a9 not found: ID does not exist"
Jul 28 17:45:13.502670 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.502631    2578 scope.go:117] "RemoveContainer" containerID="10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4"
Jul 28 17:45:13.502871 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:45:13.502853    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4\": container with ID starting with 10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4 not found: ID does not exist" containerID="10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4"
Jul 28 17:45:13.502912 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.502876    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4"} err="failed to get container status \"10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4\": rpc error: code = NotFound desc = could not find container \"10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4\": container with ID starting with 10febd2e8baada6455c1ed7c57f216c5c25b8ccda6043d1bb1cbb39dd49fb8f4 not found: ID does not exist"
Jul 28 17:45:13.600949 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:13.600926    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"]
Jul 28 17:45:13.603135 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:45:13.603108    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f5baaf_ede5_4e2c_ba04_cdeddaaec3e9.slice/crio-04ba989d22d68e54115a5cdaf21d3a0806b8248348e0deedffda697f7337ef21 WatchSource:0}: Error finding container 04ba989d22d68e54115a5cdaf21d3a0806b8248348e0deedffda697f7337ef21: Status 404 returned error can't find the container with id 04ba989d22d68e54115a5cdaf21d3a0806b8248348e0deedffda697f7337ef21
Jul 28 17:45:14.482390 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:14.482347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerStarted","Data":"c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1"}
Jul 28 17:45:14.482788 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:14.482453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerStarted","Data":"04ba989d22d68e54115a5cdaf21d3a0806b8248348e0deedffda697f7337ef21"}
Jul 28 17:45:15.362962 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:15.362918    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d2db35c-3d4b-4218-aee0-e38bcd5b3248" path="/var/lib/kubelet/pods/0d2db35c-3d4b-4218-aee0-e38bcd5b3248/volumes"
Jul 28 17:45:17.493912 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:17.493876    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerID="c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1" exitCode=0
Jul 28 17:45:17.494372 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:17.493950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerDied","Data":"c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1"}
Jul 28 17:45:18.499855 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:18.499822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerStarted","Data":"efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920"}
Jul 28 17:45:18.500314 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:18.499888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerStarted","Data":"a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66"}
Jul 28 17:45:18.500314 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:18.500081    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:18.526859 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:18.526813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podStartSLOduration=1.7647902009999998 podStartE2EDuration="5.526799728s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.762009527s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:18.524828367 +0000 UTC m=+2073.690464223" watchObservedRunningTime="2026-07-28 17:45:18.526799728 +0000 UTC m=+2073.692435561"
Jul 28 17:45:19.504118 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:19.504091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:19.505272 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:19.505244    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:45:20.507359 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:20.507330    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:45:25.512096 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:25.512069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:45:25.512626 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:25.512599    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:45:35.512799 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:35.512755    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:45:45.512684 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:45.512653    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:45:55.512936 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:45:55.512891    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:46:05.513070 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:05.513026    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:46:15.512861 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:15.512824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:46:25.513077 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:25.513043    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:46:35.513284 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:35.513253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:46:43.283629 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:43.283549    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"]
Jul 28 17:46:43.284135 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:43.283835    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" containerID="cri-o://a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66" gracePeriod=30
Jul 28 17:46:43.284135 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:43.283877    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kube-rbac-proxy" containerID="cri-o://efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920" gracePeriod=30
Jul 28 17:46:43.370626 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:43.370599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:43.825142 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:43.825110    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerID="efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920" exitCode=2
Jul 28 17:46:43.825324 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:43.825183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerDied","Data":"efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920"}
Jul 28 17:46:45.508117 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:45.508079    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 28 17:46:45.512969 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:45.512946    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:46:47.520595 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.520570    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:46:47.694557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.694524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kserve-provision-location\") pod \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") "
Jul 28 17:46:47.694739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.694573    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-cabundle-cert\") pod \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") "
Jul 28 17:46:47.694739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.694645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-proxy-tls\") pod \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") "
Jul 28 17:46:47.694739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.694674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bhzc\" (UniqueName: \"kubernetes.io/projected/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-api-access-9bhzc\") pod \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") "
Jul 28 17:46:47.694739 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.694698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-rbac-proxy-sar-config\") pod \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\" (UID: \"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9\") "
Jul 28 17:46:47.694949 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.694889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" (UID: "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:47.695115 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.695081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" (UID: "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:47.695199 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.695121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" (UID: "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:47.696723 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.696701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" (UID: "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:47.696812 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.696749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-api-access-9bhzc" (OuterVolumeSpecName: "kube-api-access-9bhzc") pod "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" (UID: "f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9"). InnerVolumeSpecName "kube-api-access-9bhzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:47.795684 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.795646    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bhzc\" (UniqueName: \"kubernetes.io/projected/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-api-access-9bhzc\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:46:47.795684 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.795675    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:46:47.795684 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.795684    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-kserve-provision-location\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:46:47.795684 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.795693    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-cabundle-cert\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:46:47.795943 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.795702    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9-proxy-tls\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:46:47.841069 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.841038    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerID="a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66" exitCode=0
Jul 28 17:46:47.841241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.841084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerDied","Data":"a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66"}
Jul 28 17:46:47.841241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.841108    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"
Jul 28 17:46:47.841241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.841142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc" event={"ID":"f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9","Type":"ContainerDied","Data":"04ba989d22d68e54115a5cdaf21d3a0806b8248348e0deedffda697f7337ef21"}
Jul 28 17:46:47.841241 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.841169    2578 scope.go:117] "RemoveContainer" containerID="efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920"
Jul 28 17:46:47.850355 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.850343    2578 scope.go:117] "RemoveContainer" containerID="a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66"
Jul 28 17:46:47.857502 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.857483    2578 scope.go:117] "RemoveContainer" containerID="c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1"
Jul 28 17:46:47.864443 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.864427    2578 scope.go:117] "RemoveContainer" containerID="efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920"
Jul 28 17:46:47.864668 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:46:47.864652    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920\": container with ID starting with efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920 not found: ID does not exist" containerID="efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920"
Jul 28 17:46:47.864724 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.864678    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920"} err="failed to get container status \"efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920\": rpc error: code = NotFound desc = could not find container \"efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920\": container with ID starting with efa6d221deb3956604855aebcbf91eb7815a95ca2fc928afaf903ac918ab4920 not found: ID does not exist"
Jul 28 17:46:47.864724 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.864696    2578 scope.go:117] "RemoveContainer" containerID="a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66"
Jul 28 17:46:47.864902 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:46:47.864883    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66\": container with ID starting with a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66 not found: ID does not exist" containerID="a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66"
Jul 28 17:46:47.864946 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.864909    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66"} err="failed to get container status \"a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66\": rpc error: code = NotFound desc = could not find container \"a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66\": container with ID starting with a8d82a4bf42512baa5afbd55f58ca98cab58df00536da4f87d2bd6e3b191cd66 not found: ID does not exist"
Jul 28 17:46:47.864946 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.864925    2578 scope.go:117] "RemoveContainer" containerID="c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1"
Jul 28 17:46:47.865193 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:46:47.865175    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1\": container with ID starting with c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1 not found: ID does not exist" containerID="c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1"
Jul 28 17:46:47.865258 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.865200    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1"} err="failed to get container status \"c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1\": rpc error: code = NotFound desc = could not find container \"c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1\": container with ID starting with c44082279f532470b96163030b131a6ad93ede96f7099e3341b40758518031f1 not found: ID does not exist"
Jul 28 17:46:47.865853 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.865836    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"]
Jul 28 17:46:47.869848 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:47.869828    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-81166-predictor-5bb94697d8-9vlsc"]
Jul 28 17:46:48.977819 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.977787    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6dskr/must-gather-rfstb"]
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978067    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kube-rbac-proxy"
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978079    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kube-rbac-proxy"
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978089    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="storage-initializer"
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978100    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="storage-initializer"
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container"
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978112    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container"
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978160    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kserve-container"
Jul 28 17:46:48.978177 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.978172    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" containerName="kube-rbac-proxy"
Jul 28 17:46:48.982280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.982260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:48.985787 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.985766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6dskr\"/\"openshift-service-ca.crt\""
Jul 28 17:46:48.985787 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.985784    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6dskr\"/\"kube-root-ca.crt\""
Jul 28 17:46:48.987290 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.987272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6dskr\"/\"default-dockercfg-b5749\""
Jul 28 17:46:48.994856 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:48.994837    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6dskr/must-gather-rfstb"]
Jul 28 17:46:49.004527 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.004507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:49.082554 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.082528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:49.104685 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.104666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rdp\" (UniqueName: \"kubernetes.io/projected/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-kube-api-access-72rdp\") pod \"must-gather-rfstb\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") " pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:49.104769 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.104707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-must-gather-output\") pod \"must-gather-rfstb\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") " pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:49.205029 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.205004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rdp\" (UniqueName: \"kubernetes.io/projected/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-kube-api-access-72rdp\") pod \"must-gather-rfstb\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") " pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:49.205130 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.205045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-must-gather-output\") pod \"must-gather-rfstb\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") " pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:49.205342 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.205324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-must-gather-output\") pod \"must-gather-rfstb\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") " pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:49.214628 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.214608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rdp\" (UniqueName: \"kubernetes.io/projected/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-kube-api-access-72rdp\") pod \"must-gather-rfstb\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") " pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:49.301528 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.301472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:46:49.363862 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.363833    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9" path="/var/lib/kubelet/pods/f1f5baaf-ede5-4e2c-ba04-cdeddaaec3e9/volumes"
Jul 28 17:46:49.427983 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.427911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6dskr/must-gather-rfstb"]
Jul 28 17:46:49.430454 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:46:49.430430    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35a84939_dbab_4a0b_86c6_b0f10cc77fb6.slice/crio-411655c65423ae862d0d2d5fb158ad791dc13ebcd4d4850a9ded6f329c24e770 WatchSource:0}: Error finding container 411655c65423ae862d0d2d5fb158ad791dc13ebcd4d4850a9ded6f329c24e770: Status 404 returned error can't find the container with id 411655c65423ae862d0d2d5fb158ad791dc13ebcd4d4850a9ded6f329c24e770
Jul 28 17:46:49.432198 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.432178    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:46:49.849838 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:49.849809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dskr/must-gather-rfstb" event={"ID":"35a84939-dbab-4a0b-86c6-b0f10cc77fb6","Type":"ContainerStarted","Data":"411655c65423ae862d0d2d5fb158ad791dc13ebcd4d4850a9ded6f329c24e770"}
Jul 28 17:46:55.878131 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:55.878089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dskr/must-gather-rfstb" event={"ID":"35a84939-dbab-4a0b-86c6-b0f10cc77fb6","Type":"ContainerStarted","Data":"7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81"}
Jul 28 17:46:55.878131 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:46:55.878136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dskr/must-gather-rfstb" event={"ID":"35a84939-dbab-4a0b-86c6-b0f10cc77fb6","Type":"ContainerStarted","Data":"198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4"}
Jul 28 17:47:17.972315 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:17.972275    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerID="198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4" exitCode=0
Jul 28 17:47:17.972718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:17.972365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6dskr/must-gather-rfstb" event={"ID":"35a84939-dbab-4a0b-86c6-b0f10cc77fb6","Type":"ContainerDied","Data":"198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4"}
Jul 28 17:47:17.972718 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:17.972664    2578 scope.go:117] "RemoveContainer" containerID="198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4"
Jul 28 17:47:18.768406 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:18.768374    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6dskr_must-gather-rfstb_35a84939-dbab-4a0b-86c6-b0f10cc77fb6/gather/0.log"
Jul 28 17:47:19.243290 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.243252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:19.387963 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.387929    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ff2pm/must-gather-mhb5f"]
Jul 28 17:47:19.390122 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.390104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.392941 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.392920    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ff2pm\"/\"openshift-service-ca.crt\""
Jul 28 17:47:19.394373 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.394349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ff2pm\"/\"default-dockercfg-9nldz\""
Jul 28 17:47:19.394493 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.394358    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ff2pm\"/\"kube-root-ca.crt\""
Jul 28 17:47:19.400229 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.400212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ff2pm/must-gather-mhb5f"]
Jul 28 17:47:19.402775 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.402750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:19.461261 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.461235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq8zf\" (UniqueName: \"kubernetes.io/projected/04eac50b-cc60-46ee-b750-f50e9bf28b3d-kube-api-access-dq8zf\") pod \"must-gather-mhb5f\" (UID: \"04eac50b-cc60-46ee-b750-f50e9bf28b3d\") " pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.461414 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.461290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04eac50b-cc60-46ee-b750-f50e9bf28b3d-must-gather-output\") pod \"must-gather-mhb5f\" (UID: \"04eac50b-cc60-46ee-b750-f50e9bf28b3d\") " pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.504689 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.504606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:19.561955 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.561933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq8zf\" (UniqueName: \"kubernetes.io/projected/04eac50b-cc60-46ee-b750-f50e9bf28b3d-kube-api-access-dq8zf\") pod \"must-gather-mhb5f\" (UID: \"04eac50b-cc60-46ee-b750-f50e9bf28b3d\") " pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.562060 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.561965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04eac50b-cc60-46ee-b750-f50e9bf28b3d-must-gather-output\") pod \"must-gather-mhb5f\" (UID: \"04eac50b-cc60-46ee-b750-f50e9bf28b3d\") " pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.562315 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.562282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04eac50b-cc60-46ee-b750-f50e9bf28b3d-must-gather-output\") pod \"must-gather-mhb5f\" (UID: \"04eac50b-cc60-46ee-b750-f50e9bf28b3d\") " pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.570857 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.570837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq8zf\" (UniqueName: \"kubernetes.io/projected/04eac50b-cc60-46ee-b750-f50e9bf28b3d-kube-api-access-dq8zf\") pod \"must-gather-mhb5f\" (UID: \"04eac50b-cc60-46ee-b750-f50e9bf28b3d\") " pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.699131 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.699105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ff2pm/must-gather-mhb5f"
Jul 28 17:47:19.824693 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.824666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ff2pm/must-gather-mhb5f"]
Jul 28 17:47:19.826478 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:47:19.826448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04eac50b_cc60_46ee_b750_f50e9bf28b3d.slice/crio-8e9e77008e2f8b2112a0fc346679ba563f7e460fd0e6685ea7b6ee299500be84 WatchSource:0}: Error finding container 8e9e77008e2f8b2112a0fc346679ba563f7e460fd0e6685ea7b6ee299500be84: Status 404 returned error can't find the container with id 8e9e77008e2f8b2112a0fc346679ba563f7e460fd0e6685ea7b6ee299500be84
Jul 28 17:47:19.980550 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:19.980515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ff2pm/must-gather-mhb5f" event={"ID":"04eac50b-cc60-46ee-b750-f50e9bf28b3d","Type":"ContainerStarted","Data":"8e9e77008e2f8b2112a0fc346679ba563f7e460fd0e6685ea7b6ee299500be84"}
Jul 28 17:47:20.988197 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:20.988156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ff2pm/must-gather-mhb5f" event={"ID":"04eac50b-cc60-46ee-b750-f50e9bf28b3d","Type":"ContainerStarted","Data":"7d52d2eb3e0fd5126088733074770a178a70114113c737321866daf4e7221126"}
Jul 28 17:47:20.988706 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:20.988215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ff2pm/must-gather-mhb5f" event={"ID":"04eac50b-cc60-46ee-b750-f50e9bf28b3d","Type":"ContainerStarted","Data":"376eea6c96f0c5f35c3bd4e6bd0f894fa490715e88c5c8b8577c4f3f25d4837a"}
Jul 28 17:47:22.018270 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:22.018243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9dlb9_30185e23-4967-457e-836b-9a335ab75449/global-pull-secret-syncer/0.log"
Jul 28 17:47:22.134854 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:22.134828    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log"
Jul 28 17:47:22.275353 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:22.275278    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-66.ec2.internal_cfcf7e829014a4c1c014b92346f6e0a6/haproxy/0.log"
Jul 28 17:47:24.207716 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.207638    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ff2pm/must-gather-mhb5f" podStartSLOduration=4.505453281 podStartE2EDuration="5.207599548s" totalImagesPullingTime="702.146267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:21.008330442 +0000 UTC m=+2196.173966274" watchObservedRunningTime="2026-07-28 17:47:24.207599548 +0000 UTC m=+2199.373235384"
Jul 28 17:47:24.208322 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.207940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6dskr/must-gather-rfstb"]
Jul 28 17:47:24.208322 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.208184    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-6dskr/must-gather-rfstb" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerName="copy" containerID="cri-o://7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81" gracePeriod=2
Jul 28 17:47:24.213015 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.212990    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6dskr/must-gather-rfstb"]
Jul 28 17:47:24.213632 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:47:24.213602    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-rfstb\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6dskr\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:47:24.584324 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.582578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6dskr_must-gather-rfstb_35a84939-dbab-4a0b-86c6-b0f10cc77fb6/copy/0.log"
Jul 28 17:47:24.588316 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.587173    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:47:24.593314 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:47:24.592513    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-rfstb\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6dskr\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:47:24.723472 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.723442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-must-gather-output\") pod \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") "
Jul 28 17:47:24.723720 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.723705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72rdp\" (UniqueName: \"kubernetes.io/projected/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-kube-api-access-72rdp\") pod \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\" (UID: \"35a84939-dbab-4a0b-86c6-b0f10cc77fb6\") "
Jul 28 17:47:24.726323 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.724898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "35a84939-dbab-4a0b-86c6-b0f10cc77fb6" (UID: "35a84939-dbab-4a0b-86c6-b0f10cc77fb6"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:24.730316 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.726702    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-kube-api-access-72rdp" (OuterVolumeSpecName: "kube-api-access-72rdp") pod "35a84939-dbab-4a0b-86c6-b0f10cc77fb6" (UID: "35a84939-dbab-4a0b-86c6-b0f10cc77fb6"). InnerVolumeSpecName "kube-api-access-72rdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:24.824916 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.824843    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-must-gather-output\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:47:24.824916 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.824880    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72rdp\" (UniqueName: \"kubernetes.io/projected/35a84939-dbab-4a0b-86c6-b0f10cc77fb6-kube-api-access-72rdp\") on node \"ip-10-0-139-66.ec2.internal\" DevicePath \"\""
Jul 28 17:47:24.846170 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.846126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:24.946751 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:24.946720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2d7hz_4da92ddf-8155-41a3-bce0-e723bc0787c0/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:25.061387 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.061360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6dskr_must-gather-rfstb_35a84939-dbab-4a0b-86c6-b0f10cc77fb6/copy/0.log"
Jul 28 17:47:25.062221 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.062194    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerID="7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81" exitCode=143
Jul 28 17:47:25.062486 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.062466    2578 scope.go:117] "RemoveContainer" containerID="7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81"
Jul 28 17:47:25.062675 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.062658    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:47:25.068845 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:47:25.068804    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-rfstb\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6dskr\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:47:25.089578 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:47:25.089540    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-rfstb\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6dskr\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:47:25.092644 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.092624    2578 scope.go:117] "RemoveContainer" containerID="198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4"
Jul 28 17:47:25.118073 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.118045    2578 scope.go:117] "RemoveContainer" containerID="7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81"
Jul 28 17:47:25.118697 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:47:25.118672    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81\": container with ID starting with 7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81 not found: ID does not exist" containerID="7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81"
Jul 28 17:47:25.118892 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.118866    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81"} err="failed to get container status \"7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81\": rpc error: code = NotFound desc = could not find container \"7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81\": container with ID starting with 7161815d0ec22a2e3c9d9433dfff4945fc66e77f39081a1cffddfc246d3d8f81 not found: ID does not exist"
Jul 28 17:47:25.119012 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.119000    2578 scope.go:117] "RemoveContainer" containerID="198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4"
Jul 28 17:47:25.119485 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:47:25.119465    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4\": container with ID starting with 198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4 not found: ID does not exist" containerID="198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4"
Jul 28 17:47:25.119635 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.119614    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4"} err="failed to get container status \"198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4\": rpc error: code = NotFound desc = could not find container \"198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4\": container with ID starting with 198c20fc5ec7bf5036b45388338bf874549f46873b731e610f8368ac764af7c4 not found: ID does not exist"
Jul 28 17:47:25.370342 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.369870    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" path="/var/lib/kubelet/pods/35a84939-dbab-4a0b-86c6-b0f10cc77fb6/volumes"
Jul 28 17:47:25.373372 ip-10-0-139-66 kubenswrapper[2578]: E0728 17:47:25.373340    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-rfstb\" is forbidden: User \"system:node:ip-10-0-139-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6dskr\": no relationship found between node 'ip-10-0-139-66.ec2.internal' and this object" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" pod="openshift-must-gather-6dskr/must-gather-rfstb"
Jul 28 17:47:25.932057 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.932024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_48ba9010-ee07-4cae-8dbd-a770f3437eec/alertmanager/0.log"
Jul 28 17:47:25.969830 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.969787    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_48ba9010-ee07-4cae-8dbd-a770f3437eec/config-reloader/0.log"
Jul 28 17:47:25.996188 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:25.996158    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_48ba9010-ee07-4cae-8dbd-a770f3437eec/kube-rbac-proxy-web/0.log"
Jul 28 17:47:26.027659 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.027631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_48ba9010-ee07-4cae-8dbd-a770f3437eec/kube-rbac-proxy/0.log"
Jul 28 17:47:26.070343 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.070235    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_48ba9010-ee07-4cae-8dbd-a770f3437eec/kube-rbac-proxy-metric/0.log"
Jul 28 17:47:26.099719 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.099695    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_48ba9010-ee07-4cae-8dbd-a770f3437eec/prom-label-proxy/0.log"
Jul 28 17:47:26.124022 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.123992    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_48ba9010-ee07-4cae-8dbd-a770f3437eec/init-config-reloader/0.log"
Jul 28 17:47:26.282907 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.282883    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f48c497b6-7hqb5_f64e18a6-bf35-4fd4-ad2b-69151ec224b4/metrics-server/0.log"
Jul 28 17:47:26.359243 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.359213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jbfr_966c33cf-d8f8-49f3-aea9-2cea30006c68/node-exporter/0.log"
Jul 28 17:47:26.383746 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.383721    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jbfr_966c33cf-d8f8-49f3-aea9-2cea30006c68/kube-rbac-proxy/0.log"
Jul 28 17:47:26.409074 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.409037    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jbfr_966c33cf-d8f8-49f3-aea9-2cea30006c68/init-textfile/0.log"
Jul 28 17:47:26.745542 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.745504    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bdfd7f79-2cb6-423d-ad0a-86aa70b53102/prometheus/0.log"
Jul 28 17:47:26.764808 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.764776    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bdfd7f79-2cb6-423d-ad0a-86aa70b53102/config-reloader/0.log"
Jul 28 17:47:26.792396 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.792224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bdfd7f79-2cb6-423d-ad0a-86aa70b53102/thanos-sidecar/0.log"
Jul 28 17:47:26.813568 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.813544    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bdfd7f79-2cb6-423d-ad0a-86aa70b53102/kube-rbac-proxy-web/0.log"
Jul 28 17:47:26.844230 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.844206    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bdfd7f79-2cb6-423d-ad0a-86aa70b53102/kube-rbac-proxy/0.log"
Jul 28 17:47:26.871507 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.871480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bdfd7f79-2cb6-423d-ad0a-86aa70b53102/kube-rbac-proxy-thanos/0.log"
Jul 28 17:47:26.895890 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.895864    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bdfd7f79-2cb6-423d-ad0a-86aa70b53102/init-config-reloader/0.log"
Jul 28 17:47:26.925527 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.925494    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-xcg59_60f8208a-8e4e-4ac6-8f55-d3bbd7984f82/prometheus-operator/0.log"
Jul 28 17:47:26.943017 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:26.942990    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-xcg59_60f8208a-8e4e-4ac6-8f55-d3bbd7984f82/kube-rbac-proxy/0.log"
Jul 28 17:47:29.541053 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:29.540988    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c8f4f955c-hw6l5_52aa805f-db0f-4931-920c-a318a64a4454/console/0.log"
Jul 28 17:47:30.256092 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.256049    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"]
Jul 28 17:47:30.256514 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.256489    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerName="copy"
Jul 28 17:47:30.256514 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.256516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerName="copy"
Jul 28 17:47:30.256699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.256552    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerName="gather"
Jul 28 17:47:30.256699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.256561    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerName="gather"
Jul 28 17:47:30.256699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.256629    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerName="gather"
Jul 28 17:47:30.256699 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.256643    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a84939-dbab-4a0b-86c6-b0f10cc77fb6" containerName="copy"
Jul 28 17:47:30.262014 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.261985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.273521 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.273494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"]
Jul 28 17:47:30.391958 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.391911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-sys\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.392149 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.392027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-proc\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.392149 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.392084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-lib-modules\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.392280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.392158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zhr\" (UniqueName: \"kubernetes.io/projected/ad2caa25-c0e4-4065-990e-754fb5661031-kube-api-access-f7zhr\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.392280 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.392216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-podres\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-lib-modules\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zhr\" (UniqueName: \"kubernetes.io/projected/ad2caa25-c0e4-4065-990e-754fb5661031-kube-api-access-f7zhr\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-podres\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-sys\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-proc\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-proc\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-lib-modules\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.493557 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-sys\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.494115 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.493612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad2caa25-c0e4-4065-990e-754fb5661031-podres\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.504871 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.502955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zhr\" (UniqueName: \"kubernetes.io/projected/ad2caa25-c0e4-4065-990e-754fb5661031-kube-api-access-f7zhr\") pod \"perf-node-gather-daemonset-rrvwd\" (UID: \"ad2caa25-c0e4-4065-990e-754fb5661031\") " pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.574402 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.574375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:30.745703 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:30.745650    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"]
Jul 28 17:47:30.747870 ip-10-0-139-66 kubenswrapper[2578]: W0728 17:47:30.747839    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podad2caa25_c0e4_4065_990e_754fb5661031.slice/crio-4931a7f4ab1bdc73f25343071a9d20dd801096bed5d195e9354e5154ad30e318 WatchSource:0}: Error finding container 4931a7f4ab1bdc73f25343071a9d20dd801096bed5d195e9354e5154ad30e318: Status 404 returned error can't find the container with id 4931a7f4ab1bdc73f25343071a9d20dd801096bed5d195e9354e5154ad30e318
Jul 28 17:47:31.091929 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.091902    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wvmd5_50e63a86-7a28-4b6e-93b9-8acefb97b80c/dns/0.log"
Jul 28 17:47:31.100330 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.100291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd" event={"ID":"ad2caa25-c0e4-4065-990e-754fb5661031","Type":"ContainerStarted","Data":"49a2d2fd6679e3a3f654b4c19d29285e70ac56fcedbed1feb01c4b291fd0aa16"}
Jul 28 17:47:31.100480 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.100353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd" event={"ID":"ad2caa25-c0e4-4065-990e-754fb5661031","Type":"ContainerStarted","Data":"4931a7f4ab1bdc73f25343071a9d20dd801096bed5d195e9354e5154ad30e318"}
Jul 28 17:47:31.100480 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.100396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:31.111108 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.111089    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wvmd5_50e63a86-7a28-4b6e-93b9-8acefb97b80c/kube-rbac-proxy/0.log"
Jul 28 17:47:31.119591 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.119551    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd" podStartSLOduration=1.119537075 podStartE2EDuration="1.119537075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:31.118257931 +0000 UTC m=+2206.283893765" watchObservedRunningTime="2026-07-28 17:47:31.119537075 +0000 UTC m=+2206.285172908"
Jul 28 17:47:31.183722 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.183692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v7x9l_f339ccd3-739e-4820-bbd6-1909f519163c/dns-node-resolver/0.log"
Jul 28 17:47:31.682530 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:31.682506    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kxnh_1a2cb418-91da-4de5-a28e-ed1cf6ffea5f/node-ca/0.log"
Jul 28 17:47:32.860048 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:32.860020    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jghvk_86b63356-74c5-46ec-a2f6-66c82cf24df0/serve-healthcheck-canary/0.log"
Jul 28 17:47:33.355904 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:33.355879    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hwb85_a1dacda5-83fd-4545-a3b1-be36741e44c7/kube-rbac-proxy/0.log"
Jul 28 17:47:33.401683 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:33.401655    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hwb85_a1dacda5-83fd-4545-a3b1-be36741e44c7/exporter/0.log"
Jul 28 17:47:33.426849 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:33.426833    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hwb85_a1dacda5-83fd-4545-a3b1-be36741e44c7/extractor/0.log"
Jul 28 17:47:35.762918 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:35.762891    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-mzsxc_15953d4a-d6af-4302-ab0a-485dad92d135/server/0.log"
Jul 28 17:47:35.994663 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:35.994634    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-c64q5_463af8a3-0241-4563-a154-ac7b83beb165/s3-init/0.log"
Jul 28 17:47:36.013242 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:36.013179    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-c64q5_463af8a3-0241-4563-a154-ac7b83beb165/download-hf-model/0.log"
Jul 28 17:47:37.114659 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:37.114631    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ff2pm/perf-node-gather-daemonset-rrvwd"
Jul 28 17:47:41.527629 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:41.527602    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxxn4_f1b81611-bb9f-4327-8b30-27fdf4699404/kube-multus-additional-cni-plugins/0.log"
Jul 28 17:47:41.550315 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:41.550286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxxn4_f1b81611-bb9f-4327-8b30-27fdf4699404/egress-router-binary-copy/0.log"
Jul 28 17:47:41.569635 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:41.569616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxxn4_f1b81611-bb9f-4327-8b30-27fdf4699404/cni-plugins/0.log"
Jul 28 17:47:41.601549 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:41.601528    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxxn4_f1b81611-bb9f-4327-8b30-27fdf4699404/bond-cni-plugin/0.log"
Jul 28 17:47:41.624378 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:41.621315    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxxn4_f1b81611-bb9f-4327-8b30-27fdf4699404/routeoverride-cni/0.log"
Jul 28 17:47:41.648489 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:41.648461    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxxn4_f1b81611-bb9f-4327-8b30-27fdf4699404/whereabouts-cni-bincopy/0.log"
Jul 28 17:47:41.671888 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:41.671851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxxn4_f1b81611-bb9f-4327-8b30-27fdf4699404/whereabouts-cni/0.log"
Jul 28 17:47:42.025979 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:42.025899    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g757g_c91bf743-8bcf-47c4-b0b5-90c088cbbd26/kube-multus/0.log"
Jul 28 17:47:42.157322 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:42.157274    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rr4wq_4fc84898-1712-4173-8e8a-3eb68293e69d/network-metrics-daemon/0.log"
Jul 28 17:47:42.175561 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:42.175535    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rr4wq_4fc84898-1712-4173-8e8a-3eb68293e69d/kube-rbac-proxy/0.log"
Jul 28 17:47:43.320785 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.320757    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/ovn-controller/0.log"
Jul 28 17:47:43.339222 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.339191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/ovn-acl-logging/0.log"
Jul 28 17:47:43.363197 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.363166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/kube-rbac-proxy-node/0.log"
Jul 28 17:47:43.387197 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.387152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 17:47:43.411955 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.411931    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/northd/0.log"
Jul 28 17:47:43.435512 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.435490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/nbdb/0.log"
Jul 28 17:47:43.462888 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.462866    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/sbdb/0.log"
Jul 28 17:47:43.517426 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:43.517404    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qdr2l_af516b76-8f7c-494c-9c49-d0b92829113d/ovnkube-controller/0.log"
Jul 28 17:47:44.521219 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:44.521183    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gf2g6_9ec6705e-d32d-48c8-851f-0c353cf1463e/network-check-target-container/0.log"
Jul 28 17:47:45.537152 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:45.537129    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fcc6d_d45de308-c906-41dd-91e0-b5d65ad0434e/iptables-alerter/0.log"
Jul 28 17:47:46.300365 ip-10-0-139-66 kubenswrapper[2578]: I0728 17:47:46.300339    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-b78kd_112143bf-d024-4e4a-837a-c6c4b6421010/tuned/0.log"