Sep 09 19:53:14.529059 ip-10-0-135-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 19:53:14.529071 ip-10-0-135-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 19:53:14.529081 ip-10-0-135-134 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 19:53:14.529410 ip-10-0-135-134 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 19:53:25.754169 ip-10-0-135-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 19:53:25.754180 ip-10-0-135-134 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6e19e17569d24313879ce61d147c5bf8 --
Sep 09 19:55:11.837712 ip-10-0-135-134 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:55:12.296209 ip-10-0-135-134 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:12.296209 ip-10-0-135-134 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:55:12.296209 ip-10-0-135-134 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:12.296209 ip-10-0-135-134 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:55:12.296209 ip-10-0-135-134 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:12.298254 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.298132    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:55:12.303138 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303119    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:12.303138 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303139    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303143    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303147    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303149    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303152    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303155    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303159    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303162    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303165    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303170    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303174    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303177    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303180    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303183    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303187    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303191    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303195    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303198    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303200    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:12.303210 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303203    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303205    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303208    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303211    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303213    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303216    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303219    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303222    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303225    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303227    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303230    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303232    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303235    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303238    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303240    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303243    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303245    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303248    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303251    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:12.303680 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303254    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303256    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303259    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303261    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303264    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303266    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303270    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303272    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303275    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303277    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303279    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303282    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303284    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303287    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303289    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303294    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303297    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303299    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303302    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303306    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:12.304130 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303310    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303312    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303315    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303318    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303320    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303323    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303325    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303328    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303330    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303333    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303335    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303337    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303340    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303343    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303347    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303349    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303352    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303355    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303373    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303376    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:12.304673 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303378    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303381    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303384    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303386    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303390    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303392    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303395    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303399    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303402    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303404    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303407    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303410    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303412    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303415    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303417    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303420    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303423    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303425    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303428    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.303430    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:12.305156 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304162    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304168    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304171    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304174    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304177    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304180    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304183    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304185    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304188    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304190    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304193    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304195    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304199    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304201    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304204    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304207    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304224    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304227    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304230    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:12.305647 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304234    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304237    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304240    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304242    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304245    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304247    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304251    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304255    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304257    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304260    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304262    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304265    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304267    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304270    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304272    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304275    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304278    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304280    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304283    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:12.306119 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304286    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304288    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304291    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304294    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304297    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304300    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304302    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304305    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304308    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304311    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304313    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304316    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304318    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304321    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304323    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304325    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304327    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304330    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304333    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:12.306603 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304335    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304338    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304341    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304343    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304346    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304348    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304350    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304353    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304355    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304373    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304376    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304378    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304385    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304388    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304391    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304394    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304397    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304399    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304403    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304407    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:12.307082 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304410    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304413    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304416    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304421    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304425    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304427    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304430    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304433    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304436    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304438    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304441    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304443    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304446    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304449    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304451    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304454    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304456    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304459    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304461    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304464    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:12.307592 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304467    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304469    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304862    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304867    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304872    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304874    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304878    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304882    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304884    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304887    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304889    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304892    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304894    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304896    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304899    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304902    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304905    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304907    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304909    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:12.308077 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304912    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304914    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304917    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304919    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304922    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304924    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304926    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304929    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304931    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304934    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304937    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304939    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304942    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304944    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304947    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304949    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304952    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304957    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304959    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304962    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:12.308658 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304965    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304968    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304970    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304973    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304975    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304978    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304982    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304985    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304988    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304991    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304995    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.304998    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305001    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305005    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305008    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305010    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305013    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305015    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305018    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:12.309177 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305021    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305024    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305027    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305029    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305032    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305035    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305038    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305041    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305043    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305046    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305050    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305052    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305054    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305057    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305060    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305063    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305066    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305068    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305071    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305073    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:12.309665 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305075    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305079    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305081    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305084    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305086    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305089    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305091    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305094    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305096    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305098    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305101    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305104    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305106    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305108    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305111    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305113    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305116    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305118    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305120    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305123    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:12.310148 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305125    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305128    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.305132    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305211    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305256    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305263    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305267    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305272    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305276    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305280    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305284    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305287    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305290    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305294    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305297    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305301    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305304    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305307    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305310    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305312    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305315    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305319    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305322    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305325    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:55:12.310636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305328    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305331    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305335    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305338    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305341    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305344    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305347    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305349    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305352    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305355    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305373    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305378    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305381    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305384    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305386    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305389    2566 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305393    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305397    2566 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305401    2566 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305403    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305406    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305411    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305414    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305417    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305420    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:55:12.311219 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305423    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305426    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305428    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305431    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305434    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305437    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305440    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305443    2566 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305447    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305449    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305453    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305456    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305458    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305461    2566 flags.go:64] FLAG: --help="false"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305464    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305467    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305470    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305474    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305477    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305480    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305483    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305486    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305489    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:55:12.311841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305492    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305495    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305498    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305501    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305504    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305508    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305511    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305514    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305516    2566 flags.go:64] FLAG: --lock-file=""
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305519    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305522    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305525    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305530    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305533    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305535    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305538    2566 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305541    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305544    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305547    2566 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305549    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305553    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305556    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305560    2566 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305563    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305566    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:55:12.312443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305569    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305575    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305578    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305581    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305584    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305594    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305596    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305600    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305603    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305607    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305612    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305615    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305619    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305622    2566 flags.go:64] FLAG: --port="10250"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305625    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305628    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-085df0dc8f9d1886e"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305631    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305634    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305637    2566 flags.go:64] FLAG: --register-node="true"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305639    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305643    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305646    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305648    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305651    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305655    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:55:12.313085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305658    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305661    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305664    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305666    2566 flags.go:64] FLAG: --runonce="false"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305669    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305672    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305675    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305677    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305680    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305684    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305687    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305690    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305693    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305696    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305698    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305701    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305704    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305707    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305710    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305717    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305719    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.305722    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306422    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306425    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306429    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:55:12.313710 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306433    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306436    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306439    2566 flags.go:64] FLAG: --v="2"
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306443    2566 flags.go:64] FLAG: --version="false"
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306447    2566 flags.go:64] FLAG: --vmodule=""
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306451    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.306454    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306580    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306583    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306587    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306590    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306593    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306596    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306598    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306601    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306603    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306608    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306610    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306614    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306616    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306619    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:12.314316 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306622    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306624    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306626    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306629    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306632    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306636    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306639    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306641    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306644    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306646    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306649    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306651    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306653    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306656    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306658    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306661    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306663    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306666    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306668    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:12.314835 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306670    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306674    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306678    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306681    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306685    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306688    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306690    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306693    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306696    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306699    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306701    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306703    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306706    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306708    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306711    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306713    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306715    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306718    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306721    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:12.315350 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306723    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306726    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306728    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306730    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306733    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306735    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306737    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306740    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306742    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306744    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306747    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306749    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306751    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306754    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306756    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306759    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306761    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306764    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306766    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:12.315942 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306769    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306771    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306773    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306776    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306778    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306781    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306784    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306786    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306788    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306792    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306795    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306798    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306801    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306804    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306806    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306808    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306811    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306813    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306815    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306818    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:12.316462 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306820    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306823    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306825    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306828    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306830    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306833    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306835    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.306838    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.307611    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.316653    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:55:12.316954 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.316672    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:55:12.317611 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.317600    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:55:12.317656 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.317612    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:55:12.317698 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317689    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:12.317698 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317698    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317701    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317705    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317717    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317720    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317723    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317726    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317728    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317731    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317733    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317736    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317738    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317741    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317744    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317747    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317750    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317752    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317755    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317758    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317760    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:12.317755 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317763    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317766    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317769    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317771    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317774    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317776    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317779    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317782    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317791    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317794    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317797    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317799    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317802    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317805    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317807    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317810    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317812    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317815    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317817    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317820    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:12.318238 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317822    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317824    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317827    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317829    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317832    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317834    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317836    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317839    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317841    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317844    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317846    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317848    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317851    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317853    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317856    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317859    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317862    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317864    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317866    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:12.318728 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317869    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317871    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317874    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317877    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317879    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317882    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317886    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317890    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317893    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317896    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317898    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317901    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317903    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317906    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317908    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317910    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317913    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317917    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317920    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:12.319204 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317923    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317926    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317929    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317932    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317934    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317937    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317940    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317943    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317946    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317949    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317951    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317954    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317956    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317959    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317961    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317964    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317967    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317969    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317972    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:12.319757 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.317974    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.317979    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318099    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318104    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318107    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318109    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318112    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318115    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318118    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318121    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318124    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318126    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318129    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318131    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318134    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:12.320235 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318136    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318139    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318141    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318144    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318146    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318148    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318151    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318154    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318156    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318159    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318161    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318164    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318166    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318169    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318172    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318174    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318177    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318179    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318182    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318184    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:12.320617 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318187    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318189    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318192    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318195    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318197    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318200    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318203    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318205    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318207    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318210    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318212    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318215    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318217    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318220    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318222    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318225    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318228    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318230    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318233    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318236    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:12.321133 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318239    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318242    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318245    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318247    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318250    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318253    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318255    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318258    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318261    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318263    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318266    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318268    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318271    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318273    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318278    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318280    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318282    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318285    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318288    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318290    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:12.321630 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318292    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318295    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318297    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318299    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318302    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318304    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318307    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318310    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318313    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318316    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318319    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318321    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318324    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318327    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318329    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318332    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318334    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318336    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318339    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:12.322135 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318341    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318344    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318346    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318350    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318353    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318372    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:12.318376    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.318382    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.318514    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.320876    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.321960    2566 server.go:1019] "Starting client certificate rotation"
Sep 09 19:55:12.322648 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.322063    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:12.322926 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.322911    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:12.325939 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.325923    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:12.331645 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.331624    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:12.341556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.341435    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:55:12.346573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.346559    2566 log.go:25] "Validated CRI v1 image API"
Sep 09 19:55:12.347412 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.347391    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:55:12.350053 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.350029    2566 fs.go:135] Filesystem UUIDs: map[2a41a9fe-83b4-4611-a33e-ef1241bd4d35:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d236e9dd-e64a-4fa4-a00c-9baf1746924a:/dev/nvme0n1p4]
Sep 09 19:55:12.350143 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.350050    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:55:12.356117 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.355993    2566 manager.go:216] Machine: {Timestamp:2026-09-09 19:55:12.354571671 +0000 UTC m=+0.382102651 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126255 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229aa643eccad4f5e63e95c8f8a5cf SystemUUID:ec229aa6-43ec-cad4-f5e6-3e95c8f8a5cf BootID:6e19e175-69d2-4313-879c-e61d147c5bf8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ef:73:25:7c:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ef:73:25:7c:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:b6:c7:94:62:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:55:12.356117 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.356103    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:55:12.356251 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.356197    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:55:12.359735 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.359707    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:55:12.359891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.359739    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:55:12.360540 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.360527    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:55:12.360579 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.360544    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:55:12.360579 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.360553    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:12.360579 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.360570    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:12.360657 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.360551    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:12.361935 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.361923    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:55:12.366562 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.366549    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:12.366678 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.366668    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:55:12.366712 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.366698    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:55:12.366712 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.366709    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:55:12.366763 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.366724    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:55:12.366763 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.366733    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:55:12.366763 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.366748    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:55:12.367130 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.367109    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:12.367172 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.367137    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:12.369569 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.369545    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:55:12.372669 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.372653    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:55:12.372742 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.372684    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:55:12.375438 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375412    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:55:12.375617 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375590    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:55:12.375617 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375619    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:55:12.375739 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375629    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:55:12.375787 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375752    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:55:12.375787 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375771    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:55:12.375870 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375798    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:55:12.375870 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375811    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:55:12.375870 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375828    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:55:12.375870 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375841    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:55:12.376022 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375883    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:55:12.376022 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.375891    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:55:12.376327 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.376312    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:55:12.376412 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.376376    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:55:12.378347 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.378313    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:55:12.378347 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.378336    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:55:12.378512 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.378396    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:55:12.380502 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.380486    2566 server.go:1268] "Started kubelet"
Sep 09 19:55:12.380602 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.380546    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:55:12.380674 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.380626    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:55:12.380770 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.380692    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:55:12.381475 ip-10-0-135-134 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:55:12.382244 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.382195    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:55:12.382783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.382767    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:55:12.385052 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.385031    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:55:12.388704 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.388687    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:12.389434 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.389420    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:55:12.389598 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.389580    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:12.390231 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390214    2566 factory.go:55] Registering systemd factory
Sep 09 19:55:12.390231 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390232    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:55:12.390376 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390237    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:55:12.390376 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390249    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:55:12.390376 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390314    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:55:12.390690 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390459    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:55:12.390690 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390467    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:55:12.390690 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390501    2566 factory.go:153] Registering CRI-O factory
Sep 09 19:55:12.390690 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390515    2566 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:55:12.390690 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390569    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:55:12.390690 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390586    2566 factory.go:103] Registering Raw factory
Sep 09 19:55:12.390690 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390598    2566 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:55:12.391005 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.390991    2566 manager.go:349] Starting recovery of all containers
Sep 09 19:55:12.391785 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.390556    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-134.ec2.internal.18d3bf353d658f94  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-134.ec2.internal,UID:ip-10-0-135-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-134.ec2.internal,},FirstTimestamp:2026-09-09 19:55:12.380456852 +0000 UTC m=+0.407987830,LastTimestamp:2026-09-09 19:55:12.380456852 +0000 UTC m=+0.407987830,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-134.ec2.internal,}"
Sep 09 19:55:12.402766 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.402701    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 19:55:12.402937 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.402914    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 19:55:12.403767 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.403745    2566 manager.go:354] Recovery completed
Sep 09 19:55:12.410482 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.410468    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:12.412404 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.412377    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:12.412477 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.412420    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:12.412477 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.412433    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:12.413048 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413026    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:55:12.413048 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413038    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:55:12.413168 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413056    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:12.413168 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413090    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:12.413312 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413295    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h84rr"
Sep 09 19:55:12.413874 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413860    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:12.413918 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413888    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:12.413918 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.413898    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:12.415269 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.415252    2566 policy_none.go:49] "None policy: Start"
Sep 09 19:55:12.415269 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.415271    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:55:12.415383 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.415282    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:55:12.417283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.417272    2566 policy_none.go:47] "Start"
Sep 09 19:55:12.421803 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.421782    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h84rr"
Sep 09 19:55:12.422841 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.422770    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-134.ec2.internal.18d3bf353f4d0ba0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-134.ec2.internal,UID:ip-10-0-135-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-134.ec2.internal,},FirstTimestamp:2026-09-09 19:55:12.41240464 +0000 UTC m=+0.439935616,LastTimestamp:2026-09-09 19:55:12.41240464 +0000 UTC m=+0.439935616,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-134.ec2.internal,}"
Sep 09 19:55:12.424930 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.424864    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-134.ec2.internal.18d3bf353f4d6001  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-134.ec2.internal,UID:ip-10-0-135-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-134.ec2.internal,},FirstTimestamp:2026-09-09 19:55:12.412426241 +0000 UTC m=+0.439957217,LastTimestamp:2026-09-09 19:55:12.412426241 +0000 UTC m=+0.439957217,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-134.ec2.internal,}"
Sep 09 19:55:12.462185 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.462141    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.463537    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.463583    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.463605    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.463643    2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.466035    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.468003    2566 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.468048    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.468062    2566 server.go:87] "Starting device plugin registration server"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.468412    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.468425    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.468502    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.468577    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.468587    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.469057    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:55:12.472311 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.469102    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:12.564862 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.564736    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal"]
Sep 09 19:55:12.564862 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.564841    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:12.565950 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.565932    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:12.566014 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.565968    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:12.566014 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.565980    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:12.568264 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.568250    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:12.568454 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.568438    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.568527 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.568473    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:12.568527 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.568503    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:12.569144 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569127    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:12.569222 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569154    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:12.569222 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569155    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:12.569222 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569164    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:12.569222 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569179    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:12.569222 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569189    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:12.569222 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569130    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:12.569510 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569236    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:12.569510 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569251    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:12.569510 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.569212    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.571550 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.571534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.571635 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.571565    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:12.572333 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.572314    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:12.572427 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.572347    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:12.572427 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.572374    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:12.580372 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.580344    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.580427 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.580381    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-134.ec2.internal\": node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:12.591512 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.591488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa3a5e79dce45b64140d388a663a929-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal\" (UID: \"efa3a5e79dce45b64140d388a663a929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.591625 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.591521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa3a5e79dce45b64140d388a663a929-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal\" (UID: \"efa3a5e79dce45b64140d388a663a929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.591625 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.591556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4c3568a288696fcc9b2d4eeb0ba150-config\") pod \"kube-apiserver-proxy-ip-10-0-135-134.ec2.internal\" (UID: \"bc4c3568a288696fcc9b2d4eeb0ba150\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.599257 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.599240    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-134.ec2.internal\" not found" node="ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.603974 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.603952    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-134.ec2.internal\" not found" node="ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.606481 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.606464    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:12.692439 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.692391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa3a5e79dce45b64140d388a663a929-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal\" (UID: \"efa3a5e79dce45b64140d388a663a929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.692439 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.692439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa3a5e79dce45b64140d388a663a929-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal\" (UID: \"efa3a5e79dce45b64140d388a663a929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.692649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.692485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4c3568a288696fcc9b2d4eeb0ba150-config\") pod \"kube-apiserver-proxy-ip-10-0-135-134.ec2.internal\" (UID: \"bc4c3568a288696fcc9b2d4eeb0ba150\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.692649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.692568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa3a5e79dce45b64140d388a663a929-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal\" (UID: \"efa3a5e79dce45b64140d388a663a929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.692649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.692573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4c3568a288696fcc9b2d4eeb0ba150-config\") pod \"kube-apiserver-proxy-ip-10-0-135-134.ec2.internal\" (UID: \"bc4c3568a288696fcc9b2d4eeb0ba150\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.692649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.692590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa3a5e79dce45b64140d388a663a929-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal\" (UID: \"efa3a5e79dce45b64140d388a663a929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.706836 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.706808    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:12.807854 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.807804    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:12.902610 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.902534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:12.909297 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:12.909273    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:12.909435 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:12.909310    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:13.010207 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.010148    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.110623 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.110590    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.211211 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.211126    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.226214 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.226189    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:13.311460 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.311426    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.321620 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.321583    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:55:13.321762 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.321743    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:13.321822 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.321770    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:13.412218 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.412181    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.427313 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.427270    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:50:12 +0000 UTC" deadline="2028-04-22 10:39:47.682934365 +0000 UTC"
Sep 09 19:55:13.427313 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.427313    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14174h44m34.255626707s"
Sep 09 19:55:13.492032 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.491972    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:13.493039 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.493025    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:13.493113 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.493054    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:13.493113 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.493064    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:13.512749 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.512722    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.523412 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.523355    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:55:13.541380 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.541330    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:13.613666 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.613626    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.714158 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:13.714127    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-134.ec2.internal\" not found"
Sep 09 19:55:13.759338 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.759101    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:13.790617 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.790574    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:13.808221 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.808188    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:13.809433 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.809408    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal"
Sep 09 19:55:13.832279 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:13.832249    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:13.853397 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:13.853355    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa3a5e79dce45b64140d388a663a929.slice/crio-cb3e1a90af881eb0fa3e3ab2e081cb9befee6515762ddc9c6dea878637be6cbb WatchSource:0}: Error finding container cb3e1a90af881eb0fa3e3ab2e081cb9befee6515762ddc9c6dea878637be6cbb: Status 404 returned error can't find the container with id cb3e1a90af881eb0fa3e3ab2e081cb9befee6515762ddc9c6dea878637be6cbb
Sep 09 19:55:14.368261 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.368225    2566 apiserver.go:52] "Watching apiserver"
Sep 09 19:55:14.378059 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.378028    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:55:14.379747 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.379716    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal","openshift-multus/multus-lkr8t","openshift-network-operator/iptables-alerter-942zz","openshift-ovn-kubernetes/ovnkube-node-4pfft","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k","openshift-cluster-node-tuning-operator/tuned-c4b25","openshift-dns/node-resolver-zwq9h","openshift-multus/multus-additional-cni-plugins-f8tlb","openshift-multus/network-metrics-daemon-s6f8r","openshift-network-diagnostics/network-check-target-qk86w","kube-system/global-pull-secret-syncer-nfgkh","kube-system/konnectivity-agent-4jc5l","kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal","openshift-image-registry/node-ca-rjq67"]
Sep 09 19:55:14.382518 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.382492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.385021 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.384994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.385543 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.385518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:55:14.385642 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.385518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:55:14.385642 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.385581    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:55:14.385821 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.385798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:55:14.385932 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.385869    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-87b2c\""
Sep 09 19:55:14.385932 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.385904    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:55:14.387523 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.387503    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:55:14.387729 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.387714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.388157 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.387973    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sfrps\""
Sep 09 19:55:14.389026 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.389004    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:14.390543 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.390178    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:55:14.390543 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.390528    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:55:14.390970 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.390797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b7ttl\""
Sep 09 19:55:14.391071 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.391053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.393022 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.392784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.394166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.393998    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bshc8\""
Sep 09 19:55:14.394166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.394001    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:55:14.394166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.394044    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:55:14.394342 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.394312    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:55:14.394342 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.394318    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:55:14.394475 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.394318    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:55:14.395661 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.395637    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:55:14.395661 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.395652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:55:14.395830 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.395649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:55:14.395889 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.395876    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:55:14.395998 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.395980    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8v26\""
Sep 09 19:55:14.396796 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.396775    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.399312 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:14.399601 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-socket-dir-parent\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.399703 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-run-netns\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.399703 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399626    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:14.399703 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-etc-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.399703 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.399703 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399691    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7hjhs\""
Sep 09 19:55:14.399983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xvg\" (UniqueName: \"kubernetes.io/projected/d3ddca81-6886-4d4e-b27b-b22c22b1e405-kube-api-access-c8xvg\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.399983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-system-cni-dir\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.399983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-cni-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.399983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc94d9fd-1190-4cdd-b62e-a7700bde4772-cni-binary-copy\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.399983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-k8s-cni-cncf-io\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.399983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-hostroot\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.400303 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399995    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.400303 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.399993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-var-lib-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.400303 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-env-overrides\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.400303 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-etc-kubernetes\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.400303 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-node-log\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovnkube-script-lib\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4npc\" (UniqueName: \"kubernetes.io/projected/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-kube-api-access-b4npc\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cnibin\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-cnibin\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-cni-bin\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-slash\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-log-socket\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbp5\" (UniqueName: \"kubernetes.io/projected/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-kube-api-access-xrbp5\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-netns\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-cni-multus\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-systemd-units\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.400721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-run-ovn-kubernetes\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-cni-bin\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovn-node-metrics-cert\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-kubelet\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h242\" (UniqueName: \"kubernetes.io/projected/dc94d9fd-1190-4cdd-b62e-a7700bde4772-kube-api-access-5h242\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-hosts-file\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-cni-netd\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovnkube-config\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-device-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-multus-certs\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-kubelet\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-ovn\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.401573 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.402450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.400983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-system-cni-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.402450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.401004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-conf-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.402450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.401035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-daemon-config\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.402450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.401093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-tmp-dir\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.402450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.401127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2zqd\" (UniqueName: \"kubernetes.io/projected/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-kube-api-access-b2zqd\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.402450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.401181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-systemd\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.402450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.401238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.402844 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.402591    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:14.402844 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.402665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:14.402844 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.402755    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:14.402844 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.402812    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:14.403747 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.403238    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:55:14.403747 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.403564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:14.403747 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.403670    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-76dxj\""
Sep 09 19:55:14.404752 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.404735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:14.404879 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.404848    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:14.408140 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.407651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.408140 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.407849    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:14.413040 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.413016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.416335 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.416311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.416820 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.416538    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4hvk\""
Sep 09 19:55:14.416820 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.416573    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:55:14.416945 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.416863    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:55:14.419369 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.419345    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:55:14.419598 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.419580    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mcv4q\""
Sep 09 19:55:14.419787 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.419771    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:55:14.419998 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.419980    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:55:14.428057 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.428035    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9khxf"
Sep 09 19:55:14.436935 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.436911    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9khxf"
Sep 09 19:55:14.469377 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.469295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal" event={"ID":"efa3a5e79dce45b64140d388a663a929","Type":"ContainerStarted","Data":"cb3e1a90af881eb0fa3e3ab2e081cb9befee6515762ddc9c6dea878637be6cbb"}
Sep 09 19:55:14.470751 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.470727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal" event={"ID":"bc4c3568a288696fcc9b2d4eeb0ba150","Type":"ContainerStarted","Data":"b46776f2fad1373f8f024d75d742054ac3c5382294e969503183edb941a521a6"}
Sep 09 19:55:14.492388 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.492357    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:55:14.501721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovnkube-config\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.501721 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a663e806-1533-458e-bc6c-87e25425a20e-konnectivity-ca\") pod \"konnectivity-agent-4jc5l\" (UID: \"a663e806-1533-458e-bc6c-87e25425a20e\") " pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-multus-certs\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-kubelet\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-ovn\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-kubernetes\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-run\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-host\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-tmp-dir\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.501948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2zqd\" (UniqueName: \"kubernetes.io/projected/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-kube-api-access-b2zqd\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-systemd\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.501993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-socket-dir-parent\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-run-netns\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-modprobe-d\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b449c6-454c-4c25-abe4-4d4decec0038-tmp\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a663e806-1533-458e-bc6c-87e25425a20e-agent-certs\") pod \"konnectivity-agent-4jc5l\" (UID: \"a663e806-1533-458e-bc6c-87e25425a20e\") " pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-system-cni-dir\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc94d9fd-1190-4cdd-b62e-a7700bde4772-cni-binary-copy\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-var-lib-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-env-overrides\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502309 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-var-lib-kubelet\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10b449c6-454c-4c25-abe4-4d4decec0038-etc-tuned\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e82940f2-81aa-47cb-8e14-dd59d258a2f7-iptables-alerter-script\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-etc-kubernetes\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovnkube-config\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e82940f2-81aa-47cb-8e14-dd59d258a2f7-host-slash\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e25d865-79c7-497b-af66-104fda1b243d-host\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e25d865-79c7-497b-af66-104fda1b243d-serviceca\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-multus-certs\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-socket-dir-parent\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-socket-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-slash\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-slash\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-var-lib-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-etc-kubernetes\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-run-netns\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.502930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-registration-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-kubelet\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502958    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-env-overrides\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.502999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-tmp-dir\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-systemd\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-log-socket\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-system-cni-dir\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503160    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysctl-conf\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-ovn\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-log-socket\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-sys\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bh5w\" (UniqueName: \"kubernetes.io/projected/10b449c6-454c-4c25-abe4-4d4decec0038-kube-api-access-2bh5w\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovn-node-metrics-cert\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.503783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-sys-fs\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc94d9fd-1190-4cdd-b62e-a7700bde4772-cni-binary-copy\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysconfig\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd2kd\" (UniqueName: \"kubernetes.io/projected/b2014487-2cbf-4ff3-a134-4a70e6b9a002-kube-api-access-cd2kd\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-kubelet\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-kubelet\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-run-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-system-cni-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-conf-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-system-cni-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.504623 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-daemon-config\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-conf-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysctl-d\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-systemd\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-lib-modules\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503932    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-dbus\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.503990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-etc-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xvg\" (UniqueName: \"kubernetes.io/projected/d3ddca81-6886-4d4e-b27b-b22c22b1e405-kube-api-access-c8xvg\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-kubelet-config\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-cni-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-etc-openvswitch\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-k8s-cni-cncf-io\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-daemon-config\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-hostroot\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.505323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504223    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-k8s-cni-cncf-io\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovnkube-script-lib\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-multus-cni-dir\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-node-log\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-hostroot\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4npc\" (UniqueName: \"kubernetes.io/projected/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-kube-api-access-b4npc\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cnibin\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-cnibin\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cnibin\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-node-log\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-cni-bin\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krscv\" (UniqueName: \"kubernetes.io/projected/e82940f2-81aa-47cb-8e14-dd59d258a2f7-kube-api-access-krscv\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-cnibin\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckmpl\" (UniqueName: \"kubernetes.io/projected/8e25d865-79c7-497b-af66-104fda1b243d-kube-api-access-ckmpl\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-cni-bin\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbp5\" (UniqueName: \"kubernetes.io/projected/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-kube-api-access-xrbp5\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.506058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-netns\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cni-binary-copy\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-cni-multus\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-systemd-units\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-run-ovn-kubernetes\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-run-netns\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-cni-bin\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-device-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-systemd-units\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc94d9fd-1190-4cdd-b62e-a7700bde4772-host-var-lib-cni-multus\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h242\" (UniqueName: \"kubernetes.io/projected/dc94d9fd-1190-4cdd-b62e-a7700bde4772-kube-api-access-5h242\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-cni-bin\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovnkube-script-lib\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-run-ovn-kubernetes\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-hosts-file\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-device-dir\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-cni-netd\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.506891 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.504948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-hosts-file\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.507636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.505006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3ddca81-6886-4d4e-b27b-b22c22b1e405-host-cni-netd\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.507636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.505140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.509234 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.509212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3ddca81-6886-4d4e-b27b-b22c22b1e405-ovn-node-metrics-cert\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.517762 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.517728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2zqd\" (UniqueName: \"kubernetes.io/projected/e9da6979-551d-44b6-bce5-7ecdc4c97ca6-kube-api-access-b2zqd\") pod \"node-resolver-zwq9h\" (UID: \"e9da6979-551d-44b6-bce5-7ecdc4c97ca6\") " pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.520636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.520608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbp5\" (UniqueName: \"kubernetes.io/projected/e4d3e68d-ec2f-4040-aacf-0b394ca7aaea-kube-api-access-xrbp5\") pod \"multus-additional-cni-plugins-f8tlb\" (UID: \"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea\") " pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.520908 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.520620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4npc\" (UniqueName: \"kubernetes.io/projected/4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b-kube-api-access-b4npc\") pod \"aws-ebs-csi-driver-node-nhc5k\" (UID: \"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.521481 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.521453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h242\" (UniqueName: \"kubernetes.io/projected/dc94d9fd-1190-4cdd-b62e-a7700bde4772-kube-api-access-5h242\") pod \"multus-lkr8t\" (UID: \"dc94d9fd-1190-4cdd-b62e-a7700bde4772\") " pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.521626 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.521607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xvg\" (UniqueName: \"kubernetes.io/projected/d3ddca81-6886-4d4e-b27b-b22c22b1e405-kube-api-access-c8xvg\") pod \"ovnkube-node-4pfft\" (UID: \"d3ddca81-6886-4d4e-b27b-b22c22b1e405\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.606040 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysconfig\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd2kd\" (UniqueName: \"kubernetes.io/projected/b2014487-2cbf-4ff3-a134-4a70e6b9a002-kube-api-access-cd2kd\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysctl-d\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-systemd\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-lib-modules\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysconfig\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-dbus\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.606238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-kubelet-config\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krscv\" (UniqueName: \"kubernetes.io/projected/e82940f2-81aa-47cb-8e14-dd59d258a2f7-kube-api-access-krscv\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckmpl\" (UniqueName: \"kubernetes.io/projected/8e25d865-79c7-497b-af66-104fda1b243d-kube-api-access-ckmpl\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-kubelet-config\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-systemd\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-lib-modules\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysctl-d\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a663e806-1533-458e-bc6c-87e25425a20e-konnectivity-ca\") pod \"konnectivity-agent-4jc5l\" (UID: \"a663e806-1533-458e-bc6c-87e25425a20e\") " pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.606402    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-kubernetes\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-dbus\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-kubernetes\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.606499    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret podName:a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:15.106472757 +0000 UTC m=+3.134003741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret") pod "global-pull-secret-syncer-nfgkh" (UID: "a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-run\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-host\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-run\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.606654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-modprobe-d\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b449c6-454c-4c25-abe4-4d4decec0038-tmp\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-host\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a663e806-1533-458e-bc6c-87e25425a20e-agent-certs\") pod \"konnectivity-agent-4jc5l\" (UID: \"a663e806-1533-458e-bc6c-87e25425a20e\") " pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-modprobe-d\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-var-lib-kubelet\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10b449c6-454c-4c25-abe4-4d4decec0038-etc-tuned\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e82940f2-81aa-47cb-8e14-dd59d258a2f7-iptables-alerter-script\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e82940f2-81aa-47cb-8e14-dd59d258a2f7-host-slash\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-var-lib-kubelet\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e25d865-79c7-497b-af66-104fda1b243d-host\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e25d865-79c7-497b-af66-104fda1b243d-host\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e25d865-79c7-497b-af66-104fda1b243d-serviceca\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysctl-conf\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-sys\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bh5w\" (UniqueName: \"kubernetes.io/projected/10b449c6-454c-4c25-abe4-4d4decec0038-kube-api-access-2bh5w\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.606930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.607078    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:14.607285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.607124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a663e806-1533-458e-bc6c-87e25425a20e-konnectivity-ca\") pod \"konnectivity-agent-4jc5l\" (UID: \"a663e806-1533-458e-bc6c-87e25425a20e\") " pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.608371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.607212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-sys\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.608371 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.607146    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:15.107126985 +0000 UTC m=+3.134657951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:14.608371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.607080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10b449c6-454c-4c25-abe4-4d4decec0038-etc-sysctl-conf\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.608371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.607434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e82940f2-81aa-47cb-8e14-dd59d258a2f7-host-slash\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.608371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.607617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e25d865-79c7-497b-af66-104fda1b243d-serviceca\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.608765 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.608740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e82940f2-81aa-47cb-8e14-dd59d258a2f7-iptables-alerter-script\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.609656 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.609630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b449c6-454c-4c25-abe4-4d4decec0038-tmp\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.609817 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.609702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10b449c6-454c-4c25-abe4-4d4decec0038-etc-tuned\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.609817 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.609779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a663e806-1533-458e-bc6c-87e25425a20e-agent-certs\") pod \"konnectivity-agent-4jc5l\" (UID: \"a663e806-1533-458e-bc6c-87e25425a20e\") " pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.613575 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.613551    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:14.613715 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.613579    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:14.613715 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.613594    2566 projected.go:196] Error preparing data for projected volume kube-api-access-8c94s for pod openshift-network-diagnostics/network-check-target-qk86w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:14.613715 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:14.613681    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s podName:4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:15.113661304 +0000 UTC m=+3.141192294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8c94s" (UniqueName: "kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s") pod "network-check-target-qk86w" (UID: "4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:14.616313 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.616290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckmpl\" (UniqueName: \"kubernetes.io/projected/8e25d865-79c7-497b-af66-104fda1b243d-kube-api-access-ckmpl\") pod \"node-ca-rjq67\" (UID: \"8e25d865-79c7-497b-af66-104fda1b243d\") " pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.616546 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.616519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krscv\" (UniqueName: \"kubernetes.io/projected/e82940f2-81aa-47cb-8e14-dd59d258a2f7-kube-api-access-krscv\") pod \"iptables-alerter-942zz\" (UID: \"e82940f2-81aa-47cb-8e14-dd59d258a2f7\") " pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.616774 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.616755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd2kd\" (UniqueName: \"kubernetes.io/projected/b2014487-2cbf-4ff3-a134-4a70e6b9a002-kube-api-access-cd2kd\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:14.616964 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.616946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bh5w\" (UniqueName: \"kubernetes.io/projected/10b449c6-454c-4c25-abe4-4d4decec0038-kube-api-access-2bh5w\") pod \"tuned-c4b25\" (UID: \"10b449c6-454c-4c25-abe4-4d4decec0038\") " pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.695917 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.695827    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f8tlb"
Sep 09 19:55:14.704022 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.703994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lkr8t"
Sep 09 19:55:14.717308 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.717271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zwq9h"
Sep 09 19:55:14.717829 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:14.717787    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc94d9fd_1190_4cdd_b62e_a7700bde4772.slice/crio-995c8eae5ef30b8866135dad5c666cfabb8810d8dd534e23cbc5d2b8f76783aa WatchSource:0}: Error finding container 995c8eae5ef30b8866135dad5c666cfabb8810d8dd534e23cbc5d2b8f76783aa: Status 404 returned error can't find the container with id 995c8eae5ef30b8866135dad5c666cfabb8810d8dd534e23cbc5d2b8f76783aa
Sep 09 19:55:14.723898 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.723876    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:14.729593 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:14.729564    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9da6979_551d_44b6_bce5_7ecdc4c97ca6.slice/crio-b57c61783ba8bc273ed64199938448142c9de9780bd376e8a85706ed2a637bd1 WatchSource:0}: Error finding container b57c61783ba8bc273ed64199938448142c9de9780bd376e8a85706ed2a637bd1: Status 404 returned error can't find the container with id b57c61783ba8bc273ed64199938448142c9de9780bd376e8a85706ed2a637bd1
Sep 09 19:55:14.731149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.731119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k"
Sep 09 19:55:14.738007 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.737826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c4b25"
Sep 09 19:55:14.738785 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:14.738757    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ddca81_6886_4d4e_b27b_b22c22b1e405.slice/crio-a886486bd9a714f248b60cbaa61068f8e87029c4d2769f56e92f02cd98fc7ec0 WatchSource:0}: Error finding container a886486bd9a714f248b60cbaa61068f8e87029c4d2769f56e92f02cd98fc7ec0: Status 404 returned error can't find the container with id a886486bd9a714f248b60cbaa61068f8e87029c4d2769f56e92f02cd98fc7ec0
Sep 09 19:55:14.745391 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.745366    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:14.746361 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.746176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-942zz"
Sep 09 19:55:14.747335 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:14.747309    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4a7f85_3848_4c0e_a0a5_de2f62d80e9b.slice/crio-681b3a6b0afa435055de256d9395a9c9066c614226c31aa8ee8c9e40c851bd9e WatchSource:0}: Error finding container 681b3a6b0afa435055de256d9395a9c9066c614226c31aa8ee8c9e40c851bd9e: Status 404 returned error can't find the container with id 681b3a6b0afa435055de256d9395a9c9066c614226c31aa8ee8c9e40c851bd9e
Sep 09 19:55:14.753906 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.753668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:14.759196 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:14.759174    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rjq67"
Sep 09 19:55:14.759456 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:14.759425    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b449c6_454c_4c25_abe4_4d4decec0038.slice/crio-f287a5f05f482957bca81bd6462cc35ecc51ca897de0ca3033589434a33b74b3 WatchSource:0}: Error finding container f287a5f05f482957bca81bd6462cc35ecc51ca897de0ca3033589434a33b74b3: Status 404 returned error can't find the container with id f287a5f05f482957bca81bd6462cc35ecc51ca897de0ca3033589434a33b74b3
Sep 09 19:55:14.772960 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:14.772883    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82940f2_81aa_47cb_8e14_dd59d258a2f7.slice/crio-60cf222f327f50eb76891516c7174b6836d4a7eede58379784bd9eb41d856791 WatchSource:0}: Error finding container 60cf222f327f50eb76891516c7174b6836d4a7eede58379784bd9eb41d856791: Status 404 returned error can't find the container with id 60cf222f327f50eb76891516c7174b6836d4a7eede58379784bd9eb41d856791
Sep 09 19:55:15.111139 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.111038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:15.111139 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.111114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:15.111453 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.111264    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:15.111453 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.111348    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret podName:a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.111312345 +0000 UTC m=+4.138843310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret") pod "global-pull-secret-syncer-nfgkh" (UID: "a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:15.111823 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.111800    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:15.111947 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.111861    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.111842855 +0000 UTC m=+4.139373821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:15.212406 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.212344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:15.212664 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.212635    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:15.212664 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.212664    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:15.212867 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.212675    2566 projected.go:196] Error preparing data for projected volume kube-api-access-8c94s for pod openshift-network-diagnostics/network-check-target-qk86w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:15.212867 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:15.212735    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s podName:4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.212716489 +0000 UTC m=+4.240247473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c94s" (UniqueName: "kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s") pod "network-check-target-qk86w" (UID: "4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:15.441669 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.441488    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:14 +0000 UTC" deadline="2028-06-05 15:52:44.299955612 +0000 UTC"
Sep 09 19:55:15.441669 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.441556    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15235h57m28.858404546s"
Sep 09 19:55:15.475590 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.475546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rjq67" event={"ID":"8e25d865-79c7-497b-af66-104fda1b243d","Type":"ContainerStarted","Data":"e4da42fedc3b9e80c57e04a78cda4a8d3aadfcecab6f3ac973fdc0c776e12635"}
Sep 09 19:55:15.477650 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.477616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4jc5l" event={"ID":"a663e806-1533-458e-bc6c-87e25425a20e","Type":"ContainerStarted","Data":"f825240430cd272075549e8828275d7d623e5cec0222d190cc05943c746ba47c"}
Sep 09 19:55:15.479616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.479575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-942zz" event={"ID":"e82940f2-81aa-47cb-8e14-dd59d258a2f7","Type":"ContainerStarted","Data":"60cf222f327f50eb76891516c7174b6836d4a7eede58379784bd9eb41d856791"}
Sep 09 19:55:15.482291 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.482186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"a886486bd9a714f248b60cbaa61068f8e87029c4d2769f56e92f02cd98fc7ec0"}
Sep 09 19:55:15.484388 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.484333    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c4b25" event={"ID":"10b449c6-454c-4c25-abe4-4d4decec0038","Type":"ContainerStarted","Data":"f287a5f05f482957bca81bd6462cc35ecc51ca897de0ca3033589434a33b74b3"}
Sep 09 19:55:15.486114 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.486056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k" event={"ID":"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b","Type":"ContainerStarted","Data":"681b3a6b0afa435055de256d9395a9c9066c614226c31aa8ee8c9e40c851bd9e"}
Sep 09 19:55:15.487514 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.487492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zwq9h" event={"ID":"e9da6979-551d-44b6-bce5-7ecdc4c97ca6","Type":"ContainerStarted","Data":"b57c61783ba8bc273ed64199938448142c9de9780bd376e8a85706ed2a637bd1"}
Sep 09 19:55:15.489487 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.489461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lkr8t" event={"ID":"dc94d9fd-1190-4cdd-b62e-a7700bde4772","Type":"ContainerStarted","Data":"995c8eae5ef30b8866135dad5c666cfabb8810d8dd534e23cbc5d2b8f76783aa"}
Sep 09 19:55:15.491040 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:15.491014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerStarted","Data":"59f69fc333f139213b4b39fb978dc285d433fff7a9ff755fcdcb8d50a5589fdd"}
Sep 09 19:55:16.121180 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.121132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:16.121359 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.121218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:16.121464 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.121377    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:16.121524 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.121517    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret podName:a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:18.12149557 +0000 UTC m=+6.149026552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret") pod "global-pull-secret-syncer-nfgkh" (UID: "a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:16.121969 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.121948    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:16.122035 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.122013    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:18.12199593 +0000 UTC m=+6.149526899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:16.222303 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.222197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:16.222502 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.222438    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:16.222502 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.222462    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:16.222502 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.222496    2566 projected.go:196] Error preparing data for projected volume kube-api-access-8c94s for pod openshift-network-diagnostics/network-check-target-qk86w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:16.222672 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.222626    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s podName:4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:18.222604217 +0000 UTC m=+6.250135187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c94s" (UniqueName: "kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s") pod "network-check-target-qk86w" (UID: "4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:16.389902 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.389869    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:16.441773 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.441730    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:14 +0000 UTC" deadline="2028-05-24 01:45:37.167254276 +0000 UTC"
Sep 09 19:55:16.441773 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.441770    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14933h50m20.725489299s"
Sep 09 19:55:16.465285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.464560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:16.465285 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.464699    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:16.465285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.465057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:16.465285 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.465128    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:16.465285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:16.465178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:16.465285 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:16.465256    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:17.500310 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:17.500201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal" event={"ID":"efa3a5e79dce45b64140d388a663a929","Type":"ContainerStarted","Data":"452cce74d49055c24c416d04c4d0f6b68816a26b178d7b791a12794814bdc44a"}
Sep 09 19:55:17.504112 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:17.504075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal" event={"ID":"bc4c3568a288696fcc9b2d4eeb0ba150","Type":"ContainerStarted","Data":"ae57a40749563d0d8ab33f40116cd49163f7e8665b963c13e84a2c39b29bd938"}
Sep 09 19:55:17.543162 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:17.543101    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-134.ec2.internal" podStartSLOduration=3.5430839880000002 podStartE2EDuration="3.543083988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:17.542880555 +0000 UTC m=+5.570411557" watchObservedRunningTime="2026-09-09 19:55:17.543083988 +0000 UTC m=+5.570614977"
Sep 09 19:55:18.139725 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:18.139677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:18.139943 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:18.139782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:18.139943 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.139825    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:18.139943 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.139921    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret podName:a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:22.139896904 +0000 UTC m=+10.167427892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret") pod "global-pull-secret-syncer-nfgkh" (UID: "a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:18.139943 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.139929    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:18.140218 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.140000    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:22.139980898 +0000 UTC m=+10.167511871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:18.240552 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:18.240509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:18.240758 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.240679    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:18.240758 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.240705    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:18.240758 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.240716    2566 projected.go:196] Error preparing data for projected volume kube-api-access-8c94s for pod openshift-network-diagnostics/network-check-target-qk86w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:18.240943 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.240767    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s podName:4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:22.240752121 +0000 UTC m=+10.268283091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c94s" (UniqueName: "kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s") pod "network-check-target-qk86w" (UID: "4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:18.464643 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:18.464548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:18.464643 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:18.464608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:18.464643 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:18.464629    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:18.464888 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.464746    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:18.465210 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.465179    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:18.465331 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:18.465269    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:20.465518 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:20.465478    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:20.466017 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:20.465484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:20.466017 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:20.465617    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:20.466017 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:20.465614    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:20.466017 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:20.465710    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:20.466017 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:20.465839    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:22.175585 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:22.175348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:22.176059 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:22.175632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:22.176059 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.175494    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:22.176059 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.175760    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.175733255 +0000 UTC m=+18.203264236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:22.176059 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.175784    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:22.176059 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.175835    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret podName:a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.175818859 +0000 UTC m=+18.203349824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret") pod "global-pull-secret-syncer-nfgkh" (UID: "a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:22.276760 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:22.276721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:22.276960 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.276924    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:22.276960 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.276953    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:22.277089 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.276967    2566 projected.go:196] Error preparing data for projected volume kube-api-access-8c94s for pod openshift-network-diagnostics/network-check-target-qk86w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:22.277140 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.277122    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s podName:4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.277102086 +0000 UTC m=+18.304633062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c94s" (UniqueName: "kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s") pod "network-check-target-qk86w" (UID: "4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:22.466502 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:22.466404    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:22.466662 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.466542    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:22.466751 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:22.466682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:22.466804 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.466776    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:22.466856 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:22.466835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:22.466980 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:22.466905    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:23.521332 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:23.520447    2566 generic.go:356] "Generic (PLEG): container finished" podID="efa3a5e79dce45b64140d388a663a929" containerID="452cce74d49055c24c416d04c4d0f6b68816a26b178d7b791a12794814bdc44a" exitCode=0
Sep 09 19:55:23.521332 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:23.520513    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal" event={"ID":"efa3a5e79dce45b64140d388a663a929","Type":"ContainerDied","Data":"452cce74d49055c24c416d04c4d0f6b68816a26b178d7b791a12794814bdc44a"}
Sep 09 19:55:24.464585 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:24.464553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:24.464585 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:24.464571    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:24.464828 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:24.464681    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:24.464828 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:24.464782    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:24.464828 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:24.464814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:24.464936 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:24.464872    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:26.463862 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:26.463821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:26.463862 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:26.463833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:26.464397 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:26.463831    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:26.464397 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:26.463965    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:26.464397 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:26.464060    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:26.464397 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:26.464171    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:28.464173 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:28.464136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:28.464582 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:28.464266    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:28.464621 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:28.464588    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:28.464672 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:28.464656    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:28.464709 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:28.464695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:28.464750 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:28.464738    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:30.240068 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:30.240012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:30.240652 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:30.240087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:30.240652 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.240189    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:30.240652 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.240188    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:30.240652 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.240253    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret podName:a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.240239358 +0000 UTC m=+34.267770338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret") pod "global-pull-secret-syncer-nfgkh" (UID: "a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:30.240652 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.240269    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.240262051 +0000 UTC m=+34.267793018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:30.341106 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:30.341061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:30.341292 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.341256    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:30.341292 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.341282    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:30.341415 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.341294    2566 projected.go:196] Error preparing data for projected volume kube-api-access-8c94s for pod openshift-network-diagnostics/network-check-target-qk86w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:30.341415 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.341387    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s podName:4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.341349776 +0000 UTC m=+34.368880741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c94s" (UniqueName: "kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s") pod "network-check-target-qk86w" (UID: "4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:30.464169 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:30.464137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:30.464342 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:30.464138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:30.464342 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.464262    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:30.464342 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:30.464138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:30.464526 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.464315    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:30.464526 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:30.464407    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:32.465421 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:32.465379    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:32.465880 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:32.465494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:32.465880 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:32.465516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:32.465880 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:32.465496    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:32.465880 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:32.465624    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:32.465880 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:32.465681    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:33.542437 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.542258    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4d3e68d-ec2f-4040-aacf-0b394ca7aaea" containerID="2611207e1c8cf66fed0c37490f4c8bd62621e7634a6e7f9743d696c88cc41176" exitCode=0
Sep 09 19:55:33.542437 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.542354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerDied","Data":"2611207e1c8cf66fed0c37490f4c8bd62621e7634a6e7f9743d696c88cc41176"}
Sep 09 19:55:33.546545 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.546509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal" event={"ID":"efa3a5e79dce45b64140d388a663a929","Type":"ContainerStarted","Data":"8f6d2b2196a98262e8a3594f23d00e44b0407531b7403ce9b1b37f157ea6379d"}
Sep 09 19:55:33.548276 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.548243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rjq67" event={"ID":"8e25d865-79c7-497b-af66-104fda1b243d","Type":"ContainerStarted","Data":"784d18ecdffa2963bb10e1a0ed91934b63553f331ae9fd6bf394a0346452cd84"}
Sep 09 19:55:33.549871 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.549814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4jc5l" event={"ID":"a663e806-1533-458e-bc6c-87e25425a20e","Type":"ContainerStarted","Data":"290bb5eed4ca8c56523b60fba918c975b842c954a823776b2121c456ac3e5c1f"}
Sep 09 19:55:33.553469 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.553445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"08d3c12c708d6f820a3bb28325ad9a8f3ac3fd73403abba9fa6024837a9cc148"}
Sep 09 19:55:33.553577 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.553523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"86e3169163bd09dddfab2510b89e9be18e9f78a630786ed6b2a248ba697259bb"}
Sep 09 19:55:33.553614 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.553576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"42b8c8f6bb38b842f8e91514cffb59e82f2d3d74755873bdd8a88b01cd686c77"}
Sep 09 19:55:33.553651 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.553624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"f13790f375f7fb5e472ca26c2af098bfe0fbb1c56b32b74c2b1cfb692d674ebc"}
Sep 09 19:55:33.553700 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.553672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"e8954fd7653576087c5f6baebe7223a0909d981086577e8d75a4d8f5f3f3180c"}
Sep 09 19:55:33.553745 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.553704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"e61e97a03a9696d7fa9e7bdf30ac1bcdcb8ddf00705e4b9399b0b42d4f7156d9"}
Sep 09 19:55:33.555147 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.555016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c4b25" event={"ID":"10b449c6-454c-4c25-abe4-4d4decec0038","Type":"ContainerStarted","Data":"14454881b60ecaf03e9b908b7ad7b905a07bfe8fa3980cb7887e9f14bbea64e7"}
Sep 09 19:55:33.556886 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.556860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k" event={"ID":"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b","Type":"ContainerStarted","Data":"bc0a8e2a5576f7574f5dee07e5d93523bdd6c523946e5c45b898a35fa4fc4682"}
Sep 09 19:55:33.558612 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.558576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zwq9h" event={"ID":"e9da6979-551d-44b6-bce5-7ecdc4c97ca6","Type":"ContainerStarted","Data":"9b1d701ae82aeb992f97816517f7da095deb4ddc8cc74534c5f3e880c59c3b8e"}
Sep 09 19:55:33.560310 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.560276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lkr8t" event={"ID":"dc94d9fd-1190-4cdd-b62e-a7700bde4772","Type":"ContainerStarted","Data":"fa0a8547879f4cbdd5fb38ffb57a7e092cf3533461a006485770e237b4de9263"}
Sep 09 19:55:33.583070 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.583005    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4jc5l" podStartSLOduration=2.681376337 podStartE2EDuration="20.582981915s" totalImagesPullingTime="17.901605578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.582460035 +0000 UTC m=+21.609991021" watchObservedRunningTime="2026-09-09 19:55:33.582981915 +0000 UTC m=+21.610512901"
Sep 09 19:55:33.618210 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.618140    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rjq67" podStartSLOduration=2.7198375649999997 podStartE2EDuration="20.618120208s" totalImagesPullingTime="17.898282643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.617905938 +0000 UTC m=+21.645436924" watchObservedRunningTime="2026-09-09 19:55:33.618120208 +0000 UTC m=+21.645651196"
Sep 09 19:55:33.618399 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.618259    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-134.ec2.internal" podStartSLOduration=19.618250036 podStartE2EDuration="19.618250036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.60143599 +0000 UTC m=+21.628966975" watchObservedRunningTime="2026-09-09 19:55:33.618250036 +0000 UTC m=+21.645781038"
Sep 09 19:55:33.636727 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.636674    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lkr8t" podStartSLOduration=3.54942608 podStartE2EDuration="21.636655757s" totalImagesPullingTime="18.087229677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.636287718 +0000 UTC m=+21.663818703" watchObservedRunningTime="2026-09-09 19:55:33.636655757 +0000 UTC m=+21.664186742"
Sep 09 19:55:33.655822 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.655764    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zwq9h" podStartSLOduration=2.6980579970000003 podStartE2EDuration="20.655749221s" totalImagesPullingTime="17.957691224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.655703471 +0000 UTC m=+21.683234482" watchObservedRunningTime="2026-09-09 19:55:33.655749221 +0000 UTC m=+21.683280233"
Sep 09 19:55:33.675187 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:33.675130    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c4b25" podStartSLOduration=2.7387381140000002 podStartE2EDuration="20.675108809s" totalImagesPullingTime="17.936370695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.674774293 +0000 UTC m=+21.702305281" watchObservedRunningTime="2026-09-09 19:55:33.675108809 +0000 UTC m=+21.702639794"
Sep 09 19:55:34.016670 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.016516    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:34.464266 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.464233    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:34.464568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.464233    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:34.464568 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:34.464401    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:34.464568 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:34.464469    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:34.464568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.464244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:34.464568 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:34.464565    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:34.479824 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.479709    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:34.016667698Z","UUID":"e657fcf1-aeee-4308-b65a-b2e2ab780dbe","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:34.482095 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.482065    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:34.482248 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.482107    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:34.565432 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.565389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-942zz" event={"ID":"e82940f2-81aa-47cb-8e14-dd59d258a2f7","Type":"ContainerStarted","Data":"72d16e8729cdda98de08a68aa43158327fa8cfab48b9c8f14d09efa94964b232"}
Sep 09 19:55:34.568480 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:34.568409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k" event={"ID":"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b","Type":"ContainerStarted","Data":"78998f71f6b7a7284a5ed23a693bd8a91b88f988c0189a5c1c307d2d56fee6c7"}
Sep 09 19:55:35.541903 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:35.541859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:35.574347 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:35.574308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k" event={"ID":"4f4a7f85-3848-4c0e-a0a5-de2f62d80e9b","Type":"ContainerStarted","Data":"3432801dcf28c8f3d599f1f780cb6bb48bcd5948beeb386c2125580cbeac81cd"}
Sep 09 19:55:35.578198 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:35.578122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"97b2164048a991746eb0e56dec1541723b51999c367fd901340198cab4a3a5e6"}
Sep 09 19:55:35.598393 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:35.598272    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-942zz" podStartSLOduration=4.686981264 podStartE2EDuration="22.598256545s" totalImagesPullingTime="17.911275281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:34.583122686 +0000 UTC m=+22.610653698" watchObservedRunningTime="2026-09-09 19:55:35.598256545 +0000 UTC m=+23.625787531"
Sep 09 19:55:35.598576 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:35.598544    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nhc5k" podStartSLOduration=2.793627508 podStartE2EDuration="22.598532943s" totalImagesPullingTime="19.804905435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:35.598157696 +0000 UTC m=+23.625688681" watchObservedRunningTime="2026-09-09 19:55:35.598532943 +0000 UTC m=+23.626063930"
Sep 09 19:55:36.464892 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:36.464854    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:36.465090 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:36.464904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:36.465090 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:36.464867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:36.465090 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:36.465005    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:36.465250 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:36.465083    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:36.465250 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:36.465160    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:38.456192 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.455941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:38.457002 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.456986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:38.464317 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.464294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:38.464317 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.464305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:38.464496 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:38.464447    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:38.464547 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:38.464482    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:38.464547 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.464512    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:38.464653 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:38.464572    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:38.588282 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.588248    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" event={"ID":"d3ddca81-6886-4d4e-b27b-b22c22b1e405","Type":"ContainerStarted","Data":"00cfde21ae889d34e082e979e054e5dd67076b5e5513f6f1e8bb0ecbbc9be5d8"}
Sep 09 19:55:38.588551 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.588524    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:38.590346 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.590322    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4d3e68d-ec2f-4040-aacf-0b394ca7aaea" containerID="706d081a3724d97bce54f1043d8476f4f7848f5e17123f825dd6ccbb8642e1ad" exitCode=0
Sep 09 19:55:38.590439 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.590409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerDied","Data":"706d081a3724d97bce54f1043d8476f4f7848f5e17123f825dd6ccbb8642e1ad"}
Sep 09 19:55:38.591403 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.591025    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4jc5l"
Sep 09 19:55:38.605449 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.605417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:38.617971 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:38.616434    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft" podStartSLOduration=7.619865766 podStartE2EDuration="25.616414838s" totalImagesPullingTime="17.996549072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:38.61531587 +0000 UTC m=+26.642846846" watchObservedRunningTime="2026-09-09 19:55:38.616414838 +0000 UTC m=+26.643945837"
Sep 09 19:55:39.592672 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:39.592635    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:39.593200 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:39.593028    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:39.608307 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:39.608283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:55:40.037015 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.036931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk86w"]
Sep 09 19:55:40.037244 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.037059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:40.037244 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:40.037156    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:40.039765 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.039740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nfgkh"]
Sep 09 19:55:40.039896 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.039833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:40.039960 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:40.039904    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:40.049203 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.049164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s6f8r"]
Sep 09 19:55:40.049390 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.049298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:40.049446 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:40.049412    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:40.598046 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.598008    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4d3e68d-ec2f-4040-aacf-0b394ca7aaea" containerID="2279ef03ff4758779fa0f0e982f8d871bca2cb55a3de93e90b60adc72dca0c89" exitCode=0
Sep 09 19:55:40.598460 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:40.598088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerDied","Data":"2279ef03ff4758779fa0f0e982f8d871bca2cb55a3de93e90b60adc72dca0c89"}
Sep 09 19:55:41.464191 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:41.463996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:41.464389 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:41.464009    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:41.464389 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:41.464293    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:41.464389 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:41.464052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:41.464389 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:41.464340    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:41.464540 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:41.464450    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:42.606504 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:42.606416    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4d3e68d-ec2f-4040-aacf-0b394ca7aaea" containerID="ddcd9586cc1e964c1a6cc0db8ba30321e348db04a942ab1c20f86a9a5019cfb7" exitCode=0
Sep 09 19:55:42.606504 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:42.606450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerDied","Data":"ddcd9586cc1e964c1a6cc0db8ba30321e348db04a942ab1c20f86a9a5019cfb7"}
Sep 09 19:55:43.464691 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:43.464650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:43.464898 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:43.464777    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:43.465025 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:43.464988    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:43.465025 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:43.465005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:43.465190 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:43.465103    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:43.465227 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:43.465192    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:45.464837 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.464788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:45.464837 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.464813    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:45.465297 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.464818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:45.465297 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:45.464907    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk86w" podUID="4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b"
Sep 09 19:55:45.465297 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:45.465015    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfgkh" podUID="a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36"
Sep 09 19:55:45.465297 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:45.465097    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6f8r" podUID="b2014487-2cbf-4ff3-a134-4a70e6b9a002"
Sep 09 19:55:45.728818 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.728734    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-134.ec2.internal" event="NodeReady"
Sep 09 19:55:45.729011 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.728889    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:55:45.764316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.764281    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7668c4ccf7-tm9rq"]
Sep 09 19:55:45.796655 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.796607    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xlnws"]
Sep 09 19:55:45.797061 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.797026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.799798 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.799769    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 19:55:45.800024 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.800006    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vz5cf\""
Sep 09 19:55:45.800416 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.800398    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 19:55:45.800681 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.800662    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 19:55:45.807410 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.806985    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 19:55:45.813236 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.813210    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qgv5q"]
Sep 09 19:55:45.813356 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.813339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:45.816267 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.816245    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-28vq4\""
Sep 09 19:55:45.816611 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.816584    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:55:45.816769 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.816746    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:55:45.829465 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.829443    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7668c4ccf7-tm9rq"]
Sep 09 19:55:45.829651 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.829484    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xlnws"]
Sep 09 19:55:45.829651 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.829512    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgv5q"]
Sep 09 19:55:45.829651 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.829591    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:45.832677 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.832657    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:55:45.833335 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.833314    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vnftv\""
Sep 09 19:55:45.833473 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.833315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:55:45.833473 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.833323    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:55:45.936633 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.936594    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:45.959678 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:45.959678 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.959927 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-trusted-ca\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.959927 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5788617e-c54d-47ca-811f-4b2fb0ec2f68-config-volume\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:45.959927 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9cv\" (UniqueName: \"kubernetes.io/projected/5788617e-c54d-47ca-811f-4b2fb0ec2f68-kube-api-access-bz9cv\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:45.959927 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w88pt\" (UniqueName: \"kubernetes.io/projected/422d7108-e3a0-41bd-9f8c-cdb796cf13ae-kube-api-access-w88pt\") pod \"ingress-canary-qgv5q\" (UID: \"422d7108-e3a0-41bd-9f8c-cdb796cf13ae\") " pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:45.959927 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-bound-sa-token\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.959927 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bz2p\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-kube-api-access-9bz2p\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.960195 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.959980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-image-registry-private-configuration\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.960195 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.960011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9b2b862-97f2-428c-94d0-910711df0eb2-ca-trust-extracted\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.960195 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.960033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-installation-pull-secrets\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:45.960195 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.960108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5788617e-c54d-47ca-811f-4b2fb0ec2f68-tmp-dir\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:45.960195 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.960137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/422d7108-e3a0-41bd-9f8c-cdb796cf13ae-cert\") pod \"ingress-canary-qgv5q\" (UID: \"422d7108-e3a0-41bd-9f8c-cdb796cf13ae\") " pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:45.960458 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:45.960201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-certificates\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.025349 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.025234    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:46.061323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5788617e-c54d-47ca-811f-4b2fb0ec2f68-config-volume\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.061323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9cv\" (UniqueName: \"kubernetes.io/projected/5788617e-c54d-47ca-811f-4b2fb0ec2f68-kube-api-access-bz9cv\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.061616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w88pt\" (UniqueName: \"kubernetes.io/projected/422d7108-e3a0-41bd-9f8c-cdb796cf13ae-kube-api-access-w88pt\") pod \"ingress-canary-qgv5q\" (UID: \"422d7108-e3a0-41bd-9f8c-cdb796cf13ae\") " pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:46.061616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-bound-sa-token\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.061616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bz2p\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-kube-api-access-9bz2p\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.061616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-image-registry-private-configuration\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.061616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9b2b862-97f2-428c-94d0-910711df0eb2-ca-trust-extracted\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.061616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-installation-pull-secrets\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.061854 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5788617e-c54d-47ca-811f-4b2fb0ec2f68-tmp-dir\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.061929 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/422d7108-e3a0-41bd-9f8c-cdb796cf13ae-cert\") pod \"ingress-canary-qgv5q\" (UID: \"422d7108-e3a0-41bd-9f8c-cdb796cf13ae\") " pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:46.061929 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-certificates\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.062030 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.062030 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.062030 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-trusted-ca\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.062158 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.062097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9b2b862-97f2-428c-94d0-910711df0eb2-ca-trust-extracted\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.062158 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.061950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5788617e-c54d-47ca-811f-4b2fb0ec2f68-config-volume\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.062158 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.062144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5788617e-c54d-47ca-811f-4b2fb0ec2f68-tmp-dir\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.062291 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.062223    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:46.062291 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.062284    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls podName:5788617e-c54d-47ca-811f-4b2fb0ec2f68 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.562262807 +0000 UTC m=+34.589793772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls") pod "dns-default-xlnws" (UID: "5788617e-c54d-47ca-811f-4b2fb0ec2f68") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:46.062493 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.062459    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:46.062493 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.062485    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7668c4ccf7-tm9rq: secret "image-registry-tls" not found
Sep 09 19:55:46.062606 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.062563    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls podName:e9b2b862-97f2-428c-94d0-910711df0eb2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.562542237 +0000 UTC m=+34.590073224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls") pod "image-registry-7668c4ccf7-tm9rq" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2") : secret "image-registry-tls" not found
Sep 09 19:55:46.062706 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.062685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-certificates\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.062937 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.062903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-trusted-ca\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.066764 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.066613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-image-registry-private-configuration\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.066873 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.066628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-installation-pull-secrets\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.066873 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.066633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/422d7108-e3a0-41bd-9f8c-cdb796cf13ae-cert\") pod \"ingress-canary-qgv5q\" (UID: \"422d7108-e3a0-41bd-9f8c-cdb796cf13ae\") " pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:46.071785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.071761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w88pt\" (UniqueName: \"kubernetes.io/projected/422d7108-e3a0-41bd-9f8c-cdb796cf13ae-kube-api-access-w88pt\") pod \"ingress-canary-qgv5q\" (UID: \"422d7108-e3a0-41bd-9f8c-cdb796cf13ae\") " pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:46.072146 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.072127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9cv\" (UniqueName: \"kubernetes.io/projected/5788617e-c54d-47ca-811f-4b2fb0ec2f68-kube-api-access-bz9cv\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.072521 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.072504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-bound-sa-token\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.073175 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.073153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bz2p\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-kube-api-access-9bz2p\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.138929 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.138889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgv5q"
Sep 09 19:55:46.264665 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.264628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:46.264862 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.264685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:46.264862 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.264798    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:46.264862 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.264799    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:46.265029 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.264884    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:18.264857629 +0000 UTC m=+66.292388607 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:46.265029 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.264905    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret podName:a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:18.264894606 +0000 UTC m=+66.292425571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret") pod "global-pull-secret-syncer-nfgkh" (UID: "a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:46.365536 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.365445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:46.365698 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.365647    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:46.365698 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.365672    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:46.365698 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.365683    2566 projected.go:196] Error preparing data for projected volume kube-api-access-8c94s for pod openshift-network-diagnostics/network-check-target-qk86w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:46.365836 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.365759    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s podName:4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:18.365743867 +0000 UTC m=+66.393274835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8c94s" (UniqueName: "kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s") pod "network-check-target-qk86w" (UID: "4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:46.566818 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.566785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:46.566818 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:46.566823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:46.567458 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.566949    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:46.567458 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.566948    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:46.567458 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.566961    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7668c4ccf7-tm9rq: secret "image-registry-tls" not found
Sep 09 19:55:46.567458 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.567051    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls podName:e9b2b862-97f2-428c-94d0-910711df0eb2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.567024543 +0000 UTC m=+35.594555528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls") pod "image-registry-7668c4ccf7-tm9rq" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2") : secret "image-registry-tls" not found
Sep 09 19:55:46.567458 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:46.567073    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls podName:5788617e-c54d-47ca-811f-4b2fb0ec2f68 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.567061168 +0000 UTC m=+35.594592132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls") pod "dns-default-xlnws" (UID: "5788617e-c54d-47ca-811f-4b2fb0ec2f68") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:47.239081 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.239044    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgv5q"]
Sep 09 19:55:47.243789 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:55:47.243761    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422d7108_e3a0_41bd_9f8c_cdb796cf13ae.slice/crio-b3e155362bcf5659b2f0c07511aa88ed373771f57cfbd6f962a7065bef15a4eb WatchSource:0}: Error finding container b3e155362bcf5659b2f0c07511aa88ed373771f57cfbd6f962a7065bef15a4eb: Status 404 returned error can't find the container with id b3e155362bcf5659b2f0c07511aa88ed373771f57cfbd6f962a7065bef15a4eb
Sep 09 19:55:47.464540 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.464508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:55:47.464677 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.464559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:55:47.464677 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.464519    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:55:47.469149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.469051    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:47.469149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.469068    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:47.469149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.469072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvgdp\""
Sep 09 19:55:47.469149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.469070    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:47.469149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.469111    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8crj\""
Sep 09 19:55:47.469511 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.469064    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:47.574544 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.574500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:47.574544 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.574546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:47.575186 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:47.574606    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:47.575186 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:47.574671    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls podName:5788617e-c54d-47ca-811f-4b2fb0ec2f68 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:49.574655816 +0000 UTC m=+37.602186779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls") pod "dns-default-xlnws" (UID: "5788617e-c54d-47ca-811f-4b2fb0ec2f68") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:47.575186 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:47.574679    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:47.575186 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:47.574693    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7668c4ccf7-tm9rq: secret "image-registry-tls" not found
Sep 09 19:55:47.575186 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:47.574765    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls podName:e9b2b862-97f2-428c-94d0-910711df0eb2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:49.574719632 +0000 UTC m=+37.602250595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls") pod "image-registry-7668c4ccf7-tm9rq" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2") : secret "image-registry-tls" not found
Sep 09 19:55:47.622724 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.622682    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4d3e68d-ec2f-4040-aacf-0b394ca7aaea" containerID="76b5c1716a55d9b3cd84407c5ffc8e0719248c5546e804b0501b9c39b71c92b7" exitCode=0
Sep 09 19:55:47.622899 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.622767    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerDied","Data":"76b5c1716a55d9b3cd84407c5ffc8e0719248c5546e804b0501b9c39b71c92b7"}
Sep 09 19:55:47.624011 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:47.623983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgv5q" event={"ID":"422d7108-e3a0-41bd-9f8c-cdb796cf13ae","Type":"ContainerStarted","Data":"b3e155362bcf5659b2f0c07511aa88ed373771f57cfbd6f962a7065bef15a4eb"}
Sep 09 19:55:48.629149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:48.629109    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4d3e68d-ec2f-4040-aacf-0b394ca7aaea" containerID="5501b186f7467c930b652e1271549bdde0c5b5bd6a74933856a17c173cdb0471" exitCode=0
Sep 09 19:55:48.629698 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:48.629184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerDied","Data":"5501b186f7467c930b652e1271549bdde0c5b5bd6a74933856a17c173cdb0471"}
Sep 09 19:55:48.862936 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:48.862738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:48.926559 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:48.926455    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:49.592694 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:49.592656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:49.592817 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:49.592712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:49.592817 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:49.592780    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.592899 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:49.592855    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls podName:5788617e-c54d-47ca-811f-4b2fb0ec2f68 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:53.592837576 +0000 UTC m=+41.620368539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls") pod "dns-default-xlnws" (UID: "5788617e-c54d-47ca-811f-4b2fb0ec2f68") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.592899 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:49.592866    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:49.592899 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:49.592886    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7668c4ccf7-tm9rq: secret "image-registry-tls" not found
Sep 09 19:55:49.593016 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:49.592938    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls podName:e9b2b862-97f2-428c-94d0-910711df0eb2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:53.592919837 +0000 UTC m=+41.620450803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls") pod "image-registry-7668c4ccf7-tm9rq" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2") : secret "image-registry-tls" not found
Sep 09 19:55:49.635349 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:49.635309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" event={"ID":"e4d3e68d-ec2f-4040-aacf-0b394ca7aaea","Type":"ContainerStarted","Data":"466aae8a2c9d147a73c37d6a952be4bec3349b7edc8c1b546234564f0c3f3329"}
Sep 09 19:55:49.660666 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:49.660612    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f8tlb" podStartSLOduration=8.536277409 podStartE2EDuration="37.660594215s" totalImagesPullingTime="29.124316806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:49.658861905 +0000 UTC m=+37.686392902" watchObservedRunningTime="2026-09-09 19:55:49.660594215 +0000 UTC m=+37.688125200"
Sep 09 19:55:49.856546 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:49.856426    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:49.956454 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:49.956425    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:50.018830 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:50.018799    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:50.639889 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:50.639852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgv5q" event={"ID":"422d7108-e3a0-41bd-9f8c-cdb796cf13ae","Type":"ContainerStarted","Data":"967cdb95bf1a9dcabff9e69e6adbfd92126084dcbad58ce4e1990baa5d09eb6c"}
Sep 09 19:55:50.659614 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:50.659556    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qgv5q" podStartSLOduration=3.233752222 podStartE2EDuration="5.659537524s" totalImagesPullingTime="2.425785302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:50.657574407 +0000 UTC m=+38.685105393" watchObservedRunningTime="2026-09-09 19:55:50.659537524 +0000 UTC m=+38.687068511"
Sep 09 19:55:50.822609 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:50.822579    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:50.890649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:50.890565    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:50.991344 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:50.991309    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:51.234665 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:51.234633    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:51.334879 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:51.334838    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:51.891743 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:51.891713    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:51.990980 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:51.990949    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:53.622602 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:53.622554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:55:53.622602 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:53.622599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:55:53.623025 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:53.622721    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:53.623025 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:53.622752    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:53.623025 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:53.622764    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7668c4ccf7-tm9rq: secret "image-registry-tls" not found
Sep 09 19:55:53.623025 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:53.622792    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls podName:5788617e-c54d-47ca-811f-4b2fb0ec2f68 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.622775528 +0000 UTC m=+49.650306492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls") pod "dns-default-xlnws" (UID: "5788617e-c54d-47ca-811f-4b2fb0ec2f68") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:53.623025 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:55:53.622814    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls podName:e9b2b862-97f2-428c-94d0-910711df0eb2 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.622801257 +0000 UTC m=+49.650332221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls") pod "image-registry-7668c4ccf7-tm9rq" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2") : secret "image-registry-tls" not found
Sep 09 19:55:54.695324 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:54.695290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:54.752531 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:54.752493    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:59.584117 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:59.584065    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:59.820036 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:59.820003    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:59.879535 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:55:59.879457    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:01.680550 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:01.680513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:56:01.680550 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:01.680549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:56:01.680992 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:01.680656    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:01.680992 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:01.680680    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:01.680992 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:01.680691    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7668c4ccf7-tm9rq: secret "image-registry-tls" not found
Sep 09 19:56:01.680992 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:01.680731    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls podName:5788617e-c54d-47ca-811f-4b2fb0ec2f68 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:17.680714074 +0000 UTC m=+65.708245041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls") pod "dns-default-xlnws" (UID: "5788617e-c54d-47ca-811f-4b2fb0ec2f68") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:01.680992 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:01.680747    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls podName:e9b2b862-97f2-428c-94d0-910711df0eb2 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:17.680740123 +0000 UTC m=+65.708271090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls") pod "image-registry-7668c4ccf7-tm9rq" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2") : secret "image-registry-tls" not found
Sep 09 19:56:03.520423 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:03.520386    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:03.649529 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:03.649489    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:04.947940 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:04.947903    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:05.050348 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:05.050315    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:07.635567 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:07.635530    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:07.701275 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:07.701243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:11.612321 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:11.612282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pfft"
Sep 09 19:56:11.770084 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:11.770047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:11.829150 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:11.829123    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:13.124297 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.124260    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2ztwg"]
Sep 09 19:56:13.127216 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.127199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.130421 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.130397    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.130725 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.130706    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.131772 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.131753    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 19:56:13.131896 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.131804    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-569zx\""
Sep 09 19:56:13.131896 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.131803    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 19:56:13.137501 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.137469    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 19:56:13.139753 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.139732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2ztwg"]
Sep 09 19:56:13.165420 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.165386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-serving-cert\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.165586 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.165435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcm2\" (UniqueName: \"kubernetes.io/projected/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-kube-api-access-6lcm2\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.165586 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.165497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-trusted-ca\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.165688 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.165595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-config\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.238277 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.238244    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"]
Sep 09 19:56:13.240622 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.240603    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:13.243646 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.243623    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.243780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.243663    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 19:56:13.243780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.243675    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lbxq8\""
Sep 09 19:56:13.243983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.243965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.257978 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.257947    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"]
Sep 09 19:56:13.266216 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.266187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcm2\" (UniqueName: \"kubernetes.io/projected/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-kube-api-access-6lcm2\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.266371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.266225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-trusted-ca\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.266371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.266285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-config\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.266371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.266312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:13.266371 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.266330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmtw\" (UniqueName: \"kubernetes.io/projected/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-kube-api-access-dfmtw\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:13.266587 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.266388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-serving-cert\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.266955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.266930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-config\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.267138 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.267116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-trusted-ca\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.269781 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.269764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-serving-cert\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.285283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.285247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcm2\" (UniqueName: \"kubernetes.io/projected/d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3-kube-api-access-6lcm2\") pod \"console-operator-85b454874b-2ztwg\" (UID: \"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3\") " pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.366854 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.366805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:13.366854 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.366851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmtw\" (UniqueName: \"kubernetes.io/projected/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-kube-api-access-dfmtw\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:13.367101 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:13.366956    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:56:13.367101 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:13.367020    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls podName:f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f nodeName:}" failed. No retries permitted until 2026-09-09 19:56:13.867005213 +0000 UTC m=+61.894536176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-s2zc9" (UID: "f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f") : secret "samples-operator-tls" not found
Sep 09 19:56:13.376080 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.376017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmtw\" (UniqueName: \"kubernetes.io/projected/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-kube-api-access-dfmtw\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:13.387722 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.387694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:13.437183 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.437149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:13.489036 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.489011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:13.712177 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.712102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2ztwg"]
Sep 09 19:56:13.715053 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:13.715028    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06edb24_e8c4_4b7b_8c87_0f3181a3a1c3.slice/crio-7ccc15ab3af3f189ceffa30afe4473a9b4237b3855aeafc7db210fe46a4c6cdd WatchSource:0}: Error finding container 7ccc15ab3af3f189ceffa30afe4473a9b4237b3855aeafc7db210fe46a4c6cdd: Status 404 returned error can't find the container with id 7ccc15ab3af3f189ceffa30afe4473a9b4237b3855aeafc7db210fe46a4c6cdd
Sep 09 19:56:13.871560 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:13.871524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:13.871808 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:13.871648    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:56:13.871808 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:13.871702    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls podName:f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f nodeName:}" failed. No retries permitted until 2026-09-09 19:56:14.871688976 +0000 UTC m=+62.899219940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-s2zc9" (UID: "f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f") : secret "samples-operator-tls" not found
Sep 09 19:56:14.700024 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:14.699976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" event={"ID":"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3","Type":"ContainerStarted","Data":"7ccc15ab3af3f189ceffa30afe4473a9b4237b3855aeafc7db210fe46a4c6cdd"}
Sep 09 19:56:14.882028 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:14.881992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:14.882191 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:14.882159    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:56:14.882277 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:14.882266    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls podName:f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.882245475 +0000 UTC m=+64.909776439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-s2zc9" (UID: "f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f") : secret "samples-operator-tls" not found
Sep 09 19:56:15.706986 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:15.706906    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/0.log"
Sep 09 19:56:15.706986 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:15.706948    2566 generic.go:356] "Generic (PLEG): container finished" podID="d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3" containerID="406455c03ab56a0325b1848e49f898c55e4379a17908ab5176fe6fa5490ef947" exitCode=255
Sep 09 19:56:15.707389 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:15.706983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" event={"ID":"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3","Type":"ContainerDied","Data":"406455c03ab56a0325b1848e49f898c55e4379a17908ab5176fe6fa5490ef947"}
Sep 09 19:56:15.707389 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:15.707258    2566 scope.go:117] "RemoveContainer" containerID="406455c03ab56a0325b1848e49f898c55e4379a17908ab5176fe6fa5490ef947"
Sep 09 19:56:16.533845 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.533807    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:16.635352 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.635320    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:16.711083 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.711055    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 19:56:16.711540 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.711491    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/0.log"
Sep 09 19:56:16.711540 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.711525    2566 generic.go:356] "Generic (PLEG): container finished" podID="d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3" containerID="b182b645cdee0b85dabd26e45cb3d02369b7526bbb36ee535085845bad2cb2c5" exitCode=255
Sep 09 19:56:16.711657 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.711550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" event={"ID":"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3","Type":"ContainerDied","Data":"b182b645cdee0b85dabd26e45cb3d02369b7526bbb36ee535085845bad2cb2c5"}
Sep 09 19:56:16.711657 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.711600    2566 scope.go:117] "RemoveContainer" containerID="406455c03ab56a0325b1848e49f898c55e4379a17908ab5176fe6fa5490ef947"
Sep 09 19:56:16.711862 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.711845    2566 scope.go:117] "RemoveContainer" containerID="b182b645cdee0b85dabd26e45cb3d02369b7526bbb36ee535085845bad2cb2c5"
Sep 09 19:56:16.712035 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:16.712017    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2ztwg_openshift-console-operator(d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3)\"" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" podUID="d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3"
Sep 09 19:56:16.896601 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:16.896502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:16.896780 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:16.896663    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:56:16.896780 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:16.896746    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls podName:f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.896725406 +0000 UTC m=+68.924256384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-s2zc9" (UID: "f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f") : secret "samples-operator-tls" not found
Sep 09 19:56:17.702176 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:17.702134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:56:17.702176 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:17.702175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:56:17.702500 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:17.702283    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:17.702500 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:17.702283    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:17.702500 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:17.702294    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7668c4ccf7-tm9rq: secret "image-registry-tls" not found
Sep 09 19:56:17.702500 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:17.702354    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls podName:5788617e-c54d-47ca-811f-4b2fb0ec2f68 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:49.702339685 +0000 UTC m=+97.729870648 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls") pod "dns-default-xlnws" (UID: "5788617e-c54d-47ca-811f-4b2fb0ec2f68") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:17.702500 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:17.702499    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls podName:e9b2b862-97f2-428c-94d0-910711df0eb2 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:49.702478388 +0000 UTC m=+97.730009358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls") pod "image-registry-7668c4ccf7-tm9rq" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2") : secret "image-registry-tls" not found
Sep 09 19:56:17.715434 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:17.715407    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 19:56:17.715773 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:17.715719    2566 scope.go:117] "RemoveContainer" containerID="b182b645cdee0b85dabd26e45cb3d02369b7526bbb36ee535085845bad2cb2c5"
Sep 09 19:56:17.715885 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:17.715866    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2ztwg_openshift-console-operator(d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3)\"" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" podUID="d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3"
Sep 09 19:56:17.979758 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:17.979718    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:18.042050 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.042021    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:18.247729 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.247649    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qnrrp"]
Sep 09 19:56:18.277291 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.277252    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnrrp"]
Sep 09 19:56:18.277465 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.277402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.280380 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.280342    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:56:18.280530 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.280349    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:56:18.280595 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.280529    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5xp9v\""
Sep 09 19:56:18.280776 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.280758    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:56:18.281671 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.281654    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:56:18.293599 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.293565    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:18.307974 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.307951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed662548-084a-408b-8fa2-44a56d36a7ae-crio-socket\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.308095 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.308018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:56:18.308095 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.308047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed662548-084a-408b-8fa2-44a56d36a7ae-data-volume\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.308095 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.308075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.308189 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.308105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.308189 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.308133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:56:18.308189 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.308152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gw9\" (UniqueName: \"kubernetes.io/projected/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-api-access-t2gw9\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.310760 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.310744    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:18.310808 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.310755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:18.319090 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.319068    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 19:56:18.319162 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.319143    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs podName:b2014487-2cbf-4ff3-a134-4a70e6b9a002 nodeName:}" failed. No retries permitted until 2026-09-09 19:57:22.319122022 +0000 UTC m=+130.346652988 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs") pod "network-metrics-daemon-s6f8r" (UID: "b2014487-2cbf-4ff3-a134-4a70e6b9a002") : secret "metrics-daemon-secret" not found
Sep 09 19:56:18.321470 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.321441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36-original-pull-secret\") pod \"global-pull-secret-syncer-nfgkh\" (UID: \"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36\") " pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:56:18.358188 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.358153    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:18.408543 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed662548-084a-408b-8fa2-44a56d36a7ae-data-volume\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.408543 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:56:18.408719 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.408719 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.408719 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gw9\" (UniqueName: \"kubernetes.io/projected/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-api-access-t2gw9\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.408719 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed662548-084a-408b-8fa2-44a56d36a7ae-crio-socket\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.408719 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.408682    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.408970 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed662548-084a-408b-8fa2-44a56d36a7ae-crio-socket\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.408970 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.408766    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:18.90874523 +0000 UTC m=+66.936276194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.408970 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.408767    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.408970 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.408832    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:18.908816589 +0000 UTC m=+66.936347553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.408970 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.408920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed662548-084a-408b-8fa2-44a56d36a7ae-data-volume\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.411532 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.411512    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:56:18.417296 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.417275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gw9\" (UniqueName: \"kubernetes.io/projected/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-api-access-t2gw9\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.422281 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.422262    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:56:18.432265 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.432244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c94s\" (UniqueName: \"kubernetes.io/projected/4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b-kube-api-access-8c94s\") pod \"network-check-target-qk86w\" (UID: \"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b\") " pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:56:18.432983 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.432967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfgkh"
Sep 09 19:56:18.718462 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.718432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8crj\""
Sep 09 19:56:18.722406 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.722378    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nfgkh"]
Sep 09 19:56:18.725426 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:18.725401    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45ab428_78e1_4ecc_8c0e_4bdddc2ddb36.slice/crio-1e0b77481a122ec8921fc906fe534c4483248971dab468f0bc7ca0b3d594734e WatchSource:0}: Error finding container 1e0b77481a122ec8921fc906fe534c4483248971dab468f0bc7ca0b3d594734e: Status 404 returned error can't find the container with id 1e0b77481a122ec8921fc906fe534c4483248971dab468f0bc7ca0b3d594734e
Sep 09 19:56:18.725514 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.725459    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:56:18.912765 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.912722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.912765 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.912765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:18.913031 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.912868    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.913031 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.912948    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.912927533 +0000 UTC m=+67.940458513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.913031 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.912874    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.913031 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:18.913005    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.912994129 +0000 UTC m=+67.940525093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.993882 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:18.993842    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk86w"]
Sep 09 19:56:18.998382 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:18.998332    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddf9d9b_99c2_4e54_8d05_a9df36d22b2b.slice/crio-9b26c1ca615a1406d037ca6db97f5e070a2997bde56e9a1c544e756372e79d5d WatchSource:0}: Error finding container 9b26c1ca615a1406d037ca6db97f5e070a2997bde56e9a1c544e756372e79d5d: Status 404 returned error can't find the container with id 9b26c1ca615a1406d037ca6db97f5e070a2997bde56e9a1c544e756372e79d5d
Sep 09 19:56:19.061427 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:19.061320    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:19.721836 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:19.721763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk86w" event={"ID":"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b","Type":"ContainerStarted","Data":"9b26c1ca615a1406d037ca6db97f5e070a2997bde56e9a1c544e756372e79d5d"}
Sep 09 19:56:19.723154 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:19.723123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nfgkh" event={"ID":"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36","Type":"ContainerStarted","Data":"1e0b77481a122ec8921fc906fe534c4483248971dab468f0bc7ca0b3d594734e"}
Sep 09 19:56:19.922218 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:19.922181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:19.922218 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:19.922226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:19.922505 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:19.922320    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:19.922505 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:19.922418    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.922398495 +0000 UTC m=+69.949929478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:19.922505 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:19.922419    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:19.922505 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:19.922461    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.922450548 +0000 UTC m=+69.949981512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:20.410951 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:20.410916    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zwq9h_e9da6979-551d-44b6-bce5-7ecdc4c97ca6/dns-node-resolver/0.log"
Sep 09 19:56:20.930395 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:20.930336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:20.930819 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:20.930489    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:56:20.930819 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:20.930559    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls podName:f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f nodeName:}" failed. No retries permitted until 2026-09-09 19:56:28.930544024 +0000 UTC m=+76.958074988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-s2zc9" (UID: "f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f") : secret "samples-operator-tls" not found
Sep 09 19:56:21.035667 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:21.035627    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.136947 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:21.136911    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.410925 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:21.410899    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rjq67_8e25d865-79c7-497b-af66-104fda1b243d/node-ca/0.log"
Sep 09 19:56:21.810579 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:21.810463    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.870048 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:21.870013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.939002 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:21.938962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:21.939448 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:21.939011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:21.939448 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:21.939105    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:21.939448 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:21.939173    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:21.939448 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:21.939189    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:25.939168502 +0000 UTC m=+73.966699484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:21.939448 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:21.939227    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:25.93921118 +0000 UTC m=+73.966742166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:22.211339 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.211252    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgv5q_422d7108-e3a0-41bd-9f8c-cdb796cf13ae/serve-healthcheck-canary/0.log"
Sep 09 19:56:22.309341 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.309310    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-ppnvw"]
Sep 09 19:56:22.312388 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.312349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.315080 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.315054    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 19:56:22.316477 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.316452    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zjnv7\""
Sep 09 19:56:22.316477 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.316467    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 19:56:22.316658 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.316479    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 19:56:22.316763 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.316745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 19:56:22.321865 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.321801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-ppnvw"]
Sep 09 19:56:22.343665 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.343635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/61a26f8c-4538-48ce-af6a-259640ed9d3c-signing-cabundle\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.343861 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.343687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/61a26f8c-4538-48ce-af6a-259640ed9d3c-signing-key\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.343861 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.343762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xbwf\" (UniqueName: \"kubernetes.io/projected/61a26f8c-4538-48ce-af6a-259640ed9d3c-kube-api-access-4xbwf\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.346206 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.346183    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:22.402957 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.402927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:22.444630 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.444599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/61a26f8c-4538-48ce-af6a-259640ed9d3c-signing-key\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.444887 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.444652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbwf\" (UniqueName: \"kubernetes.io/projected/61a26f8c-4538-48ce-af6a-259640ed9d3c-kube-api-access-4xbwf\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.444887 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.444741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/61a26f8c-4538-48ce-af6a-259640ed9d3c-signing-cabundle\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.445515 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.445483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/61a26f8c-4538-48ce-af6a-259640ed9d3c-signing-cabundle\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.447771 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.447747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/61a26f8c-4538-48ce-af6a-259640ed9d3c-signing-key\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.453902 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.453875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbwf\" (UniqueName: \"kubernetes.io/projected/61a26f8c-4538-48ce-af6a-259640ed9d3c-kube-api-access-4xbwf\") pod \"service-ca-75d99c484-ppnvw\" (UID: \"61a26f8c-4538-48ce-af6a-259640ed9d3c\") " pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.624524 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.624485    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-ppnvw"
Sep 09 19:56:22.734408 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.734348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk86w" event={"ID":"4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b","Type":"ContainerStarted","Data":"67dca06b4bc8957ad0fe9382e0d9b6462a450decd31328e0a04d4f6ad1f984b3"}
Sep 09 19:56:22.734595 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.734566    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:56:22.750870 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.750696    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qk86w" podStartSLOduration=66.762488242 podStartE2EDuration="1m9.750678126s" totalImagesPullingTime="2.988189884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:22.749919802 +0000 UTC m=+70.777450791" watchObservedRunningTime="2026-09-09 19:56:22.750678126 +0000 UTC m=+70.778209112"
Sep 09 19:56:22.915116 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:22.915027    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-ppnvw"]
Sep 09 19:56:22.918187 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:22.918156    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a26f8c_4538_48ce_af6a_259640ed9d3c.slice/crio-90e952713ee7cc583848878309f5ff985a11587886c95103102718d9fc98f24f WatchSource:0}: Error finding container 90e952713ee7cc583848878309f5ff985a11587886c95103102718d9fc98f24f: Status 404 returned error can't find the container with id 90e952713ee7cc583848878309f5ff985a11587886c95103102718d9fc98f24f
Sep 09 19:56:23.248035 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:23.247999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:23.438127 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:23.438088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:23.438295 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:23.438153    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:23.438577 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:23.438559    2566 scope.go:117] "RemoveContainer" containerID="b182b645cdee0b85dabd26e45cb3d02369b7526bbb36ee535085845bad2cb2c5"
Sep 09 19:56:23.438753 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:23.438733    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2ztwg_openshift-console-operator(d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3)\"" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" podUID="d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3"
Sep 09 19:56:23.739019 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:23.738978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-ppnvw" event={"ID":"61a26f8c-4538-48ce-af6a-259640ed9d3c","Type":"ContainerStarted","Data":"90e952713ee7cc583848878309f5ff985a11587886c95103102718d9fc98f24f"}
Sep 09 19:56:24.743481 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:24.743447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nfgkh" event={"ID":"a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36","Type":"ContainerStarted","Data":"6b4e5fdfd1c13707d0169890921c72e7563f5afabe3baa25fe6e0154f48b7284"}
Sep 09 19:56:24.760271 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:24.760218    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nfgkh" podStartSLOduration=66.058346257 podStartE2EDuration="1m11.760201138s" totalImagesPullingTime="5.701854881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:24.759988988 +0000 UTC m=+72.787519975" watchObservedRunningTime="2026-09-09 19:56:24.760201138 +0000 UTC m=+72.787732123"
Sep 09 19:56:25.982409 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:25.982347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:25.982844 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:25.982428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:25.982844 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:25.982507    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:25.982844 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:25.982595    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.982574291 +0000 UTC m=+82.010105258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:25.982844 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:25.982647    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:25.982844 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:25.982722    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.982703468 +0000 UTC m=+82.010234435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:26.120298 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:26.120258    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:26.749897 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:26.749859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-ppnvw" event={"ID":"61a26f8c-4538-48ce-af6a-259640ed9d3c","Type":"ContainerStarted","Data":"b4f18b193f16d288c0b4bc5a4a83ad47da688caca3586b9e95e3d3ed698a45ba"}
Sep 09 19:56:26.770104 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:26.770049    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-ppnvw" podStartSLOduration=1.341130686 podStartE2EDuration="4.77002667s" totalImagesPullingTime="3.428895984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:26.769488907 +0000 UTC m=+74.797019894" watchObservedRunningTime="2026-09-09 19:56:26.77002667 +0000 UTC m=+74.797557657"
Sep 09 19:56:28.107349 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:28.107310    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:28.216986 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:28.216944    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:28.309116 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:28.309074    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:28.409135 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:28.409032    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:28.476031 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:28.475984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:28.781688 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:28.781635    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:29.008914 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:29.008873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:29.011535 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:29.011502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-s2zc9\" (UID: \"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:29.148732 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:29.148629    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"
Sep 09 19:56:29.423398 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:29.423279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9"]
Sep 09 19:56:29.562217 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:29.562182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:29.759336 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:29.759296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9" event={"ID":"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f","Type":"ContainerStarted","Data":"5a36757ffb94e5ff8f64c75b101a329c4e44b93f3c598a021716b20fc831f592"}
Sep 09 19:56:29.963268 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:29.963235    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:30.067651 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:30.067568    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:30.363968 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:30.363860    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:30.762275 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:30.762230    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:30.974705 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:30.974671    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:31.367511 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:31.367470    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:31.478792 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:31.478757    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:31.766463 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:31.766419    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:31.868019 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:31.867967    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:32.175433 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:32.175343    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:33.024961 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:33.024913    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:33.107532 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:33.107501    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:33.369034 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:33.368888    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:33.773902 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:33.773864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9" event={"ID":"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f","Type":"ContainerStarted","Data":"b257a61a7364dc24f8f96554e7f74db659584467d464933e08dffb3a6c5133af"}
Sep 09 19:56:33.774085 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:33.773961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9" event={"ID":"f1987fbe-14ea-4ddf-b445-4c8bcbbbd39f","Type":"ContainerStarted","Data":"6d35f3263e309466dc22891828803435c376e045088466b8b162c61ee1f93186"}
Sep 09 19:56:33.793051 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:33.792978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-s2zc9" podStartSLOduration=17.554709746 podStartE2EDuration="20.792961112s" totalImagesPullingTime="3.238251366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:33.792266626 +0000 UTC m=+81.819797611" watchObservedRunningTime="2026-09-09 19:56:33.792961112 +0000 UTC m=+81.820492143"
Sep 09 19:56:34.050400 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:34.050280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:34.050400 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:34.050325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:34.050897 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:34.050522    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:34.050897 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:34.050605    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls podName:ed662548-084a-408b-8fa2-44a56d36a7ae nodeName:}" failed. No retries permitted until 2026-09-09 19:56:50.050585158 +0000 UTC m=+98.078116122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qnrrp" (UID: "ed662548-084a-408b-8fa2-44a56d36a7ae") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:34.050897 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:34.050863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed662548-084a-408b-8fa2-44a56d36a7ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:34.166397 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:34.166349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:34.345777 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:34.345680    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:34.410547 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:34.410513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:34.762875 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:34.762839    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:35.464642 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:35.464608    2566 scope.go:117] "RemoveContainer" containerID="b182b645cdee0b85dabd26e45cb3d02369b7526bbb36ee535085845bad2cb2c5"
Sep 09 19:56:35.783339 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:35.783309    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 19:56:35.783543 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:35.783386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" event={"ID":"d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3","Type":"ContainerStarted","Data":"1272dffe1cb56765ba52d4ba47a24648b077336817ef8f31a5f9cb3e6f24dc91"}
Sep 09 19:56:35.783697 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:35.783669    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:35.806212 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:35.806169    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-2ztwg" podStartSLOduration=21.112601413 podStartE2EDuration="22.806153968s" totalImagesPullingTime="1.693552555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:35.804833646 +0000 UTC m=+83.832364644" watchObservedRunningTime="2026-09-09 19:56:35.806153968 +0000 UTC m=+83.833684954"
Sep 09 19:56:35.843093 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:35.843062    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-2ztwg"
Sep 09 19:56:36.585109 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:36.585070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:36.649182 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:36.649145    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:36.741151 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:36.741121    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:36.807339 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:36.807294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:36.970698 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:36.970670    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:37.368122 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:37.368035    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:37.730947 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:37.730886    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:38.047986 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:38.047898    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:38.149226 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:38.149197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:38.362194 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:38.362115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:38.565454 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:38.565420    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:38.964481 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:38.964443    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:39.169342 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:39.169307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:40.569127 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:40.569091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:41.931547 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:41.931513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:41.987830 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:41.987797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:44.664605 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:44.664568    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:45.878924 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.878887    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"]
Sep 09 19:56:45.885150 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.885125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:45.891715 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.891554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:45.892955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.892907    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 19:56:45.893074 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.892983    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:45.893166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.893083    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:45.893394 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.893378    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cdj9q\""
Sep 09 19:56:45.894968 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.894948    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"]
Sep 09 19:56:45.982018 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.981975    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-ctbm8"]
Sep 09 19:56:45.986207 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.986182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-ctbm8"
Sep 09 19:56:45.990026 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.990000    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zb9ht\""
Sep 09 19:56:45.990026 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.990014    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 19:56:45.990273 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:45.990258    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 19:56:46.001449 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.001419    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-ctbm8"]
Sep 09 19:56:46.038618 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.038579    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:46.042616 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.042596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/845d90df-3d48-4695-a529-aa6c3e964157-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58c89c8868-79wz8\" (UID: \"845d90df-3d48-4695-a529-aa6c3e964157\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:46.042707 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.042627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wn82\" (UniqueName: \"kubernetes.io/projected/845d90df-3d48-4695-a529-aa6c3e964157-kube-api-access-7wn82\") pod \"managed-serviceaccount-addon-agent-58c89c8868-79wz8\" (UID: \"845d90df-3d48-4695-a529-aa6c3e964157\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:46.114605 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.114559    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:46.143268 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.143181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/845d90df-3d48-4695-a529-aa6c3e964157-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58c89c8868-79wz8\" (UID: \"845d90df-3d48-4695-a529-aa6c3e964157\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:46.143268 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.143217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49s9l\" (UniqueName: \"kubernetes.io/projected/85dabe45-811b-4f26-80e0-e6c00f8a8048-kube-api-access-49s9l\") pod \"downloads-85d98c998f-ctbm8\" (UID: \"85dabe45-811b-4f26-80e0-e6c00f8a8048\") " pod="openshift-console/downloads-85d98c998f-ctbm8"
Sep 09 19:56:46.143268 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.143241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wn82\" (UniqueName: \"kubernetes.io/projected/845d90df-3d48-4695-a529-aa6c3e964157-kube-api-access-7wn82\") pod \"managed-serviceaccount-addon-agent-58c89c8868-79wz8\" (UID: \"845d90df-3d48-4695-a529-aa6c3e964157\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:46.145972 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.145942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/845d90df-3d48-4695-a529-aa6c3e964157-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58c89c8868-79wz8\" (UID: \"845d90df-3d48-4695-a529-aa6c3e964157\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:46.158138 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.158105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wn82\" (UniqueName: \"kubernetes.io/projected/845d90df-3d48-4695-a529-aa6c3e964157-kube-api-access-7wn82\") pod \"managed-serviceaccount-addon-agent-58c89c8868-79wz8\" (UID: \"845d90df-3d48-4695-a529-aa6c3e964157\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:46.204450 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.204411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"
Sep 09 19:56:46.244508 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.244437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49s9l\" (UniqueName: \"kubernetes.io/projected/85dabe45-811b-4f26-80e0-e6c00f8a8048-kube-api-access-49s9l\") pod \"downloads-85d98c998f-ctbm8\" (UID: \"85dabe45-811b-4f26-80e0-e6c00f8a8048\") " pod="openshift-console/downloads-85d98c998f-ctbm8"
Sep 09 19:56:46.257055 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.257031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49s9l\" (UniqueName: \"kubernetes.io/projected/85dabe45-811b-4f26-80e0-e6c00f8a8048-kube-api-access-49s9l\") pod \"downloads-85d98c998f-ctbm8\" (UID: \"85dabe45-811b-4f26-80e0-e6c00f8a8048\") " pod="openshift-console/downloads-85d98c998f-ctbm8"
Sep 09 19:56:46.294817 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.294783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-ctbm8"
Sep 09 19:56:46.479377 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.479335    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8"]
Sep 09 19:56:46.482270 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:46.482244    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod845d90df_3d48_4695_a529_aa6c3e964157.slice/crio-bdf2f0667a9adf9d6a710a9336fa590ed2e151f539f54e0b88337e9bc3de07a6 WatchSource:0}: Error finding container bdf2f0667a9adf9d6a710a9336fa590ed2e151f539f54e0b88337e9bc3de07a6: Status 404 returned error can't find the container with id bdf2f0667a9adf9d6a710a9336fa590ed2e151f539f54e0b88337e9bc3de07a6
Sep 09 19:56:46.566684 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.566659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-ctbm8"]
Sep 09 19:56:46.569046 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:46.568994    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dabe45_811b_4f26_80e0_e6c00f8a8048.slice/crio-05be824e25d6219d824cbc05b163717e2dd6092d4a2590153257dfe1732c03bd WatchSource:0}: Error finding container 05be824e25d6219d824cbc05b163717e2dd6092d4a2590153257dfe1732c03bd: Status 404 returned error can't find the container with id 05be824e25d6219d824cbc05b163717e2dd6092d4a2590153257dfe1732c03bd
Sep 09 19:56:46.821236 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.821136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8" event={"ID":"845d90df-3d48-4695-a529-aa6c3e964157","Type":"ContainerStarted","Data":"bdf2f0667a9adf9d6a710a9336fa590ed2e151f539f54e0b88337e9bc3de07a6"}
Sep 09 19:56:46.822217 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.822183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-ctbm8" event={"ID":"85dabe45-811b-4f26-80e0-e6c00f8a8048","Type":"ContainerStarted","Data":"05be824e25d6219d824cbc05b163717e2dd6092d4a2590153257dfe1732c03bd"}
Sep 09 19:56:46.926515 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.926483    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:46.992379 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:46.992324    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:47.068902 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:47.068859    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:47.931149 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:47.931115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:48.030045 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.030006    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:48.100329 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.100293    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"]
Sep 09 19:56:48.103878 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.103850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"
Sep 09 19:56:48.107770 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.107516    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9hkhf\""
Sep 09 19:56:48.107770 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.107607    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 19:56:48.111827 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.111795    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"]
Sep 09 19:56:48.122022 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.121938    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:48.263289 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.263236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-j9gk7\" (UID: \"ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"
Sep 09 19:56:48.364342 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.364302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-j9gk7\" (UID: \"ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"
Sep 09 19:56:48.367599 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.367530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-j9gk7\" (UID: \"ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"
Sep 09 19:56:48.394757 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.394700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:48.423949 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:48.423907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"
Sep 09 19:56:49.720203 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.720175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"]
Sep 09 19:56:49.723523 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:49.723497    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7cb7b9_a4f9_416a_8ead_62cf7c089b8d.slice/crio-cc9a51f5204479afcaddb92d8e818b54907784daf62ed815f73dba3945196ac6 WatchSource:0}: Error finding container cc9a51f5204479afcaddb92d8e818b54907784daf62ed815f73dba3945196ac6: Status 404 returned error can't find the container with id cc9a51f5204479afcaddb92d8e818b54907784daf62ed815f73dba3945196ac6
Sep 09 19:56:49.778969 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.778933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:56:49.778969 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.778968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:56:49.781875 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.781838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5788617e-c54d-47ca-811f-4b2fb0ec2f68-metrics-tls\") pod \"dns-default-xlnws\" (UID: \"5788617e-c54d-47ca-811f-4b2fb0ec2f68\") " pod="openshift-dns/dns-default-xlnws"
Sep 09 19:56:49.782021 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.781908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"image-registry-7668c4ccf7-tm9rq\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") " pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:56:49.837053 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.837008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7" event={"ID":"ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d","Type":"ContainerStarted","Data":"cc9a51f5204479afcaddb92d8e818b54907784daf62ed815f73dba3945196ac6"}
Sep 09 19:56:49.838775 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.838746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8" event={"ID":"845d90df-3d48-4695-a529-aa6c3e964157","Type":"ContainerStarted","Data":"f66566942306469eafd08c3b22b5056ee31ef28ffcd49ca6fc0f8cab948d61de"}
Sep 09 19:56:49.855863 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:49.855794    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58c89c8868-79wz8" podStartSLOduration=1.824735448 podStartE2EDuration="4.855770221s" totalImagesPullingTime="3.031034773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:49.855162642 +0000 UTC m=+97.882693629" watchObservedRunningTime="2026-09-09 19:56:49.855770221 +0000 UTC m=+97.883301220"
Sep 09 19:56:50.013237 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.013147    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vz5cf\""
Sep 09 19:56:50.020785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.020755    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:56:50.029161 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.029132    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-28vq4\""
Sep 09 19:56:50.036631 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.036599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xlnws"
Sep 09 19:56:50.082270 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.082235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:50.087686 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.087658    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed662548-084a-408b-8fa2-44a56d36a7ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qnrrp\" (UID: \"ed662548-084a-408b-8fa2-44a56d36a7ae\") " pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:50.309271 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.309178    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7668c4ccf7-tm9rq"]
Sep 09 19:56:50.313574 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:50.313314    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b2b862_97f2_428c_94d0_910711df0eb2.slice/crio-5bb99fd73c5820c6d44a0b66dbae7b7be194308443442b2c0d2db29fba00823b WatchSource:0}: Error finding container 5bb99fd73c5820c6d44a0b66dbae7b7be194308443442b2c0d2db29fba00823b: Status 404 returned error can't find the container with id 5bb99fd73c5820c6d44a0b66dbae7b7be194308443442b2c0d2db29fba00823b
Sep 09 19:56:50.327541 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.326315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xlnws"]
Sep 09 19:56:50.329110 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.329050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:50.330610 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:50.330579    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5788617e_c54d_47ca_811f_4b2fb0ec2f68.slice/crio-8a6caa9337ee968c2dce3ea8b37ade8da7b5e84ab41a3628951c2cdd5b909e1a WatchSource:0}: Error finding container 8a6caa9337ee968c2dce3ea8b37ade8da7b5e84ab41a3628951c2cdd5b909e1a: Status 404 returned error can't find the container with id 8a6caa9337ee968c2dce3ea8b37ade8da7b5e84ab41a3628951c2cdd5b909e1a
Sep 09 19:56:50.386681 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.386657    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qnrrp"
Sep 09 19:56:50.430797 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.430718    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:50.686771 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.686632    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qnrrp"]
Sep 09 19:56:50.826689 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:56:50.826652    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded662548_084a_408b_8fa2_44a56d36a7ae.slice/crio-c4a9b8556a3d0f063a38e57bb93397900cb84794e50e24759aaac03c5b59f5f4 WatchSource:0}: Error finding container c4a9b8556a3d0f063a38e57bb93397900cb84794e50e24759aaac03c5b59f5f4: Status 404 returned error can't find the container with id c4a9b8556a3d0f063a38e57bb93397900cb84794e50e24759aaac03c5b59f5f4
Sep 09 19:56:50.844827 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.844795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnrrp" event={"ID":"ed662548-084a-408b-8fa2-44a56d36a7ae","Type":"ContainerStarted","Data":"c4a9b8556a3d0f063a38e57bb93397900cb84794e50e24759aaac03c5b59f5f4"}
Sep 09 19:56:50.847919 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.847883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xlnws" event={"ID":"5788617e-c54d-47ca-811f-4b2fb0ec2f68","Type":"ContainerStarted","Data":"8a6caa9337ee968c2dce3ea8b37ade8da7b5e84ab41a3628951c2cdd5b909e1a"}
Sep 09 19:56:50.851227 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.850723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq" event={"ID":"e9b2b862-97f2-428c-94d0-910711df0eb2","Type":"ContainerStarted","Data":"1d078d7c1bfd90a8a36c60c4e4f1ebdb19fc6aa4530987cf3db85b63f5ac6c0f"}
Sep 09 19:56:50.851227 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.850769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq" event={"ID":"e9b2b862-97f2-428c-94d0-910711df0eb2","Type":"ContainerStarted","Data":"5bb99fd73c5820c6d44a0b66dbae7b7be194308443442b2c0d2db29fba00823b"}
Sep 09 19:56:50.871388 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:50.871253    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq" podStartSLOduration=97.871237981 podStartE2EDuration="1m37.871237981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:50.870856999 +0000 UTC m=+98.898387986" watchObservedRunningTime="2026-09-09 19:56:50.871237981 +0000 UTC m=+98.898768963"
Sep 09 19:56:51.856553 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:51.856504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7" event={"ID":"ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d","Type":"ContainerStarted","Data":"4a8feba16d99605871ccaa83b97ba825b51a5908457c6a38f6c9e329a9c67085"}
Sep 09 19:56:51.857027 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:51.856793    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"
Sep 09 19:56:51.859505 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:51.859478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnrrp" event={"ID":"ed662548-084a-408b-8fa2-44a56d36a7ae","Type":"ContainerStarted","Data":"c5509ba8b4392ede55f911307166cfcf4f5198e6589bd5b33e2d9cfe989bb5ad"}
Sep 09 19:56:51.859658 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:51.859537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnrrp" event={"ID":"ed662548-084a-408b-8fa2-44a56d36a7ae","Type":"ContainerStarted","Data":"29cca196b1aa1b9b866b60630833c5d5f473fa45701f695a097a97bae82f0b0f"}
Sep 09 19:56:51.859726 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:51.859663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:56:51.863042 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:51.863018    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7"
Sep 09 19:56:51.873597 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:51.873539    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-j9gk7" podStartSLOduration=2.7195192280000002 podStartE2EDuration="3.873522182s" totalImagesPullingTime="1.154002954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:51.871874692 +0000 UTC m=+99.899405714" watchObservedRunningTime="2026-09-09 19:56:51.873522182 +0000 UTC m=+99.901053167"
Sep 09 19:56:52.131325 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:52.131257    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:52.679723 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:52.679614    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:52.732820 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:52.732784    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:52.865377 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:52.865319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xlnws" event={"ID":"5788617e-c54d-47ca-811f-4b2fb0ec2f68","Type":"ContainerStarted","Data":"0ad4970f1c3fce56e72e5953fdd1e60ab07d8ea8ca3eb613e0f560475abfd550"}
Sep 09 19:56:52.865749 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:52.865403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xlnws" event={"ID":"5788617e-c54d-47ca-811f-4b2fb0ec2f68","Type":"ContainerStarted","Data":"59adc5b4be97da63de3cfa26560bfc2dc94f238c2bfcaa6f8d76c6ccd7d40543"}
Sep 09 19:56:52.865749 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:52.865436    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xlnws"
Sep 09 19:56:52.884187 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:52.884132    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xlnws" podStartSLOduration=65.763444391 podStartE2EDuration="1m7.884113678s" totalImagesPullingTime="2.120669287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:52.883077179 +0000 UTC m=+100.910608164" watchObservedRunningTime="2026-09-09 19:56:52.884113678 +0000 UTC m=+100.911644663"
Sep 09 19:56:53.742254 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:53.742217    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qk86w"
Sep 09 19:56:54.256733 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:54.256684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:54.877846 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:54.877758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qnrrp" event={"ID":"ed662548-084a-408b-8fa2-44a56d36a7ae","Type":"ContainerStarted","Data":"04bc034599a95e3c9b1ebbdf671a350663ec9ec983ccb83cdae3cf974c3c8c33"}
Sep 09 19:56:54.905420 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:54.905335    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qnrrp" podStartSLOduration=33.263189016 podStartE2EDuration="36.905316719s" totalImagesPullingTime="3.642127703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:54.904705967 +0000 UTC m=+102.932236975" watchObservedRunningTime="2026-09-09 19:56:54.905316719 +0000 UTC m=+102.932847730"
Sep 09 19:56:55.531343 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:55.531299    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:55.590599 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:55.590552    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.097251 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.097197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.197835 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.197770    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.298771 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.298728    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.517469 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.517432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.611407 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.611352    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"]
Sep 09 19:56:56.616110 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.616045    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.637372 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.637324    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5mbwq"]
Sep 09 19:56:56.637706 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.637677    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.641526 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.641205    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 19:56:56.641526 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.641472    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 19:56:56.641732 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.641709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 19:56:56.642779 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.642756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vrcqm\""
Sep 09 19:56:56.642995 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.642959    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 19:56:56.644345 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.643170    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 19:56:56.653453 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.653419    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"]
Sep 09 19:56:56.653599 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.653478    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"]
Sep 09 19:56:56.654201 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.653772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.657318 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.657221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 19:56:56.661962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.660833    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f5wvt\""
Sep 09 19:56:56.661962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.661091    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 19:56:56.662234 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.662112    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 19:56:56.663971 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.663704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"]
Sep 09 19:56:56.663971 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.663848    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.666507 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.666245    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 19:56:56.667386 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.666775    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 19:56:56.667386 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.667040    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 19:56:56.667386 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.667221    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zrv9b\""
Sep 09 19:56:56.745313 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-wtmp\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745531 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.745531 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nd9q\" (UniqueName: \"kubernetes.io/projected/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-kube-api-access-8nd9q\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745531 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-tls\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745531 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-textfile\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745737 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-metrics-client-ca\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745737 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-root\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745737 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f3e40-bf9e-415d-86b9-9142516823e3-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.745737 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-sys\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745737 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745737 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.745955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.745955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.745776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrjb\" (UniqueName: \"kubernetes.io/projected/8f8f3e40-bf9e-415d-86b9-9142516823e3-kube-api-access-7xrjb\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.767418 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.767321    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.825810 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.825727    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:56.847112 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-wtmp\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847326 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2884be2-bbc7-4511-a78d-9933573cee86-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.847326 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.847326 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nd9q\" (UniqueName: \"kubernetes.io/projected/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-kube-api-access-8nd9q\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847326 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-tls\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847326 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-wtmp\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847326 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-textfile\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847326 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-metrics-client-ca\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-root\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f3e40-bf9e-415d-86b9-9142516823e3-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-sys\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2884be2-bbc7-4511-a78d-9933573cee86-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r722s\" (UniqueName: \"kubernetes.io/projected/b2884be2-bbc7-4511-a78d-9933573cee86-kube-api-access-r722s\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.847682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrjb\" (UniqueName: \"kubernetes.io/projected/8f8f3e40-bf9e-415d-86b9-9142516823e3-kube-api-access-7xrjb\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.847728 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:56.847719    2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 19:56:56.848350 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:56.847782    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-tls podName:8f8f3e40-bf9e-415d-86b9-9142516823e3 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:57.34776257 +0000 UTC m=+105.375293548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-d26g4" (UID: "8f8f3e40-bf9e-415d-86b9-9142516823e3") : secret "openshift-state-metrics-tls" not found
Sep 09 19:56:56.848476 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.848450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.848530 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.848498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-root\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.848660 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:56.848642    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 19:56:56.848736 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:56.848704    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-tls podName:99e876b6-dc17-4c45-bf5d-2ee473f6bf73 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:57.348687058 +0000 UTC m=+105.376218021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-tls") pod "node-exporter-5mbwq" (UID: "99e876b6-dc17-4c45-bf5d-2ee473f6bf73") : secret "node-exporter-tls" not found
Sep 09 19:56:56.849011 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.848983    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-textfile\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.849200 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.849060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-sys\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.849306 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.849204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f3e40-bf9e-415d-86b9-9142516823e3-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.849445 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.849346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-metrics-client-ca\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.851089 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.851064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.851938 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.851838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.857801 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.857733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nd9q\" (UniqueName: \"kubernetes.io/projected/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-kube-api-access-8nd9q\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:56.859171 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.859083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrjb\" (UniqueName: \"kubernetes.io/projected/8f8f3e40-bf9e-415d-86b9-9142516823e3-kube-api-access-7xrjb\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:56.948193 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.948140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2884be2-bbc7-4511-a78d-9933573cee86-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.948193 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.948196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.948490 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.948262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r722s\" (UniqueName: \"kubernetes.io/projected/b2884be2-bbc7-4511-a78d-9933573cee86-kube-api-access-r722s\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.948490 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.948317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.948490 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.948348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.948490 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.948484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2884be2-bbc7-4511-a78d-9933573cee86-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.949017 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.948985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.949260 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.949241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2884be2-bbc7-4511-a78d-9933573cee86-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.949863 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.949791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2884be2-bbc7-4511-a78d-9933573cee86-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.952083 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.952060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.952628 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.952607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2884be2-bbc7-4511-a78d-9933573cee86-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.958802 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.958776    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r722s\" (UniqueName: \"kubernetes.io/projected/b2884be2-bbc7-4511-a78d-9933573cee86-kube-api-access-r722s\") pod \"kube-state-metrics-74cb57df7-vxnm9\" (UID: \"b2884be2-bbc7-4511-a78d-9933573cee86\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.979760 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.979722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"
Sep 09 19:56:56.990225 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:56.990188    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:57.096446 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.096348    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:57.190951 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.190908    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:57.351917 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.351829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:57.351917 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.351875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-tls\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:57.355001 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.354966    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99e876b6-dc17-4c45-bf5d-2ee473f6bf73-node-exporter-tls\") pod \"node-exporter-5mbwq\" (UID: \"99e876b6-dc17-4c45-bf5d-2ee473f6bf73\") " pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:57.355107 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.355073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f8f3e40-bf9e-415d-86b9-9142516823e3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-d26g4\" (UID: \"8f8f3e40-bf9e-415d-86b9-9142516823e3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:57.519549 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.519513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:57.553498 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.553461    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"
Sep 09 19:56:57.569850 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.569815    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5mbwq"
Sep 09 19:56:57.608694 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.608606    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:57.612841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.612812    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:56:57.627804 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.627770    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.633787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.633828    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7cc46\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.633861    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634108    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634294    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634310    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634478    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634609    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634636    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634658    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 19:56:57.635785 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.634614    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 19:56:57.755336 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755336 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhmf\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-kube-api-access-5fhmf\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-config-out\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755793 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755793 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755793 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-web-config\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755793 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.755793 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.755725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856449 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhmf\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-kube-api-access-5fhmf\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-config-out\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.856955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.856836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-web-config\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.859963 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.858824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.859963 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:57.858967    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle podName:365dd335-87a8-4c08-a25a-74db86c680ff nodeName:}" failed. No retries permitted until 2026-09-09 19:56:58.358944272 +0000 UTC m=+106.386475255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff") : configmap references non-existent config key: ca-bundle.crt
Sep 09 19:56:57.860185 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.860155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.860300 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:57.860276    2566 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 19:56:57.860397 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:56:57.860383    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls podName:365dd335-87a8-4c08-a25a-74db86c680ff nodeName:}" failed. No retries permitted until 2026-09-09 19:56:58.360339626 +0000 UTC m=+106.387870595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff") : secret "alertmanager-main-tls" not found
Sep 09 19:56:57.860535 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.860509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.861401 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.861304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-web-config\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.861499 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.861460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.862241 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.862183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.862823 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.862792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.863248 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.863208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.863248 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.863237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-config-out\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.864269 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.864246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.869163 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.869121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhmf\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-kube-api-access-5fhmf\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:57.922564 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:57.922521    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:58.362240 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:58.362196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:58.362449 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:58.362256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:58.363246 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:58.363218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:58.365061 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:58.365038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:58.541271 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:58.541235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:58.552592 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:58.552554    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:58.640260 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:58.640136    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:59.050227 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:59.050182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:59.117142 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:59.117106    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:59.499574 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:59.499541    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:59.599962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:59.599929    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:59.955177 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:56:59.955011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:00.057896 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.057854    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:00.842183 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.842142    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5475f4dd97-ldk2k"]
Sep 09 19:57:00.845223 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.845196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.849133 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.849100    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 19:57:00.849297 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.849273    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 19:57:00.850666 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.850638    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 19:57:00.850666 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.850649    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-10dd9559gbgr5\""
Sep 09 19:57:00.850900 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.850691    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 19:57:00.850900 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.850650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-w6xkn\""
Sep 09 19:57:00.855658 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.855621    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5475f4dd97-ldk2k"]
Sep 09 19:57:00.885068 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.885037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed369a29-24d0-4290-ad74-c7b9cab47ba1-metrics-server-audit-profiles\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.885256 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.885140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed369a29-24d0-4290-ad74-c7b9cab47ba1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.885256 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.885160    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-secret-metrics-server-tls\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.885256 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.885185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-client-ca-bundle\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.885256 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.885225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-secret-metrics-server-client-certs\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.885256 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.885247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv7k\" (UniqueName: \"kubernetes.io/projected/ed369a29-24d0-4290-ad74-c7b9cab47ba1-kube-api-access-vjv7k\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.885499 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.885278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed369a29-24d0-4290-ad74-c7b9cab47ba1-audit-log\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.986059 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.986023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed369a29-24d0-4290-ad74-c7b9cab47ba1-metrics-server-audit-profiles\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.986556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.986122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed369a29-24d0-4290-ad74-c7b9cab47ba1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.986556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.986150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-secret-metrics-server-tls\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.986556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.986177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-client-ca-bundle\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.986556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.986212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-secret-metrics-server-client-certs\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.986556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.986238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv7k\" (UniqueName: \"kubernetes.io/projected/ed369a29-24d0-4290-ad74-c7b9cab47ba1-kube-api-access-vjv7k\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.986556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.986268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed369a29-24d0-4290-ad74-c7b9cab47ba1-audit-log\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.987660 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.987238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed369a29-24d0-4290-ad74-c7b9cab47ba1-metrics-server-audit-profiles\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.987660 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.987441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed369a29-24d0-4290-ad74-c7b9cab47ba1-audit-log\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.987877 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.987836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed369a29-24d0-4290-ad74-c7b9cab47ba1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.989925 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.989903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-secret-metrics-server-client-certs\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.990248 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.990223    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-client-ca-bundle\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.990348 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.990328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed369a29-24d0-4290-ad74-c7b9cab47ba1-secret-metrics-server-tls\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:00.995708 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:00.995681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv7k\" (UniqueName: \"kubernetes.io/projected/ed369a29-24d0-4290-ad74-c7b9cab47ba1-kube-api-access-vjv7k\") pod \"metrics-server-5475f4dd97-ldk2k\" (UID: \"ed369a29-24d0-4290-ad74-c7b9cab47ba1\") " pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:01.158692 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.158610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:01.339874 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.339833    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:01.388307 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.388268    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9f8fcccd-g4x64"]
Sep 09 19:57:01.391166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.391131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.394424 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.394398    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 19:57:01.394684 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.394665    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 19:57:01.394777 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.394719    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nvnxj\""
Sep 09 19:57:01.394834 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.394667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 19:57:01.395818 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.395798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 19:57:01.395975 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.395949    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 19:57:01.400733 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.400711    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 19:57:01.405586 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.405562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9f8fcccd-g4x64"]
Sep 09 19:57:01.427542 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.427451    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:01.490306 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.490269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-oauth-serving-cert\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.490521 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.490327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-service-ca\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.490521 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.490399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qf9\" (UniqueName: \"kubernetes.io/projected/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-kube-api-access-g9qf9\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.490521 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.490484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-oauth-config\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.490521 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.490509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-config\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.490744 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.490554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-serving-cert\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.490744 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.490667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-trusted-ca-bundle\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.591841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.591803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-trusted-ca-bundle\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.591841 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.591851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-oauth-serving-cert\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.592108 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.591895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-service-ca\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.592108 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.591923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qf9\" (UniqueName: \"kubernetes.io/projected/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-kube-api-access-g9qf9\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.592108 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.591996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-oauth-config\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.592108 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.592022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-config\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.592108 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.592080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-serving-cert\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.593800 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.592621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-oauth-serving-cert\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.593800 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.593197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-service-ca\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.593800 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.593207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-trusted-ca-bundle\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.593800 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.593735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-config\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.595264 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.595218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-serving-cert\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.595703 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.595678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-oauth-config\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.601454 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.601426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qf9\" (UniqueName: \"kubernetes.io/projected/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-kube-api-access-g9qf9\") pod \"console-5d9f8fcccd-g4x64\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") " pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.704674 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.704524    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:01.763351 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:01.763314    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:02.676885 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:02.676852    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:02.873032 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:02.872995    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xlnws"
Sep 09 19:57:02.878675 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:02.878642    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:04.107242 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:57:04.107042    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e876b6_dc17_4c45_bf5d_2ee473f6bf73.slice/crio-a9577af8834aad6e6365887ed07ff2bf8efd556bcc89b6e7d94ce9e267b29e89 WatchSource:0}: Error finding container a9577af8834aad6e6365887ed07ff2bf8efd556bcc89b6e7d94ce9e267b29e89: Status 404 returned error can't find the container with id a9577af8834aad6e6365887ed07ff2bf8efd556bcc89b6e7d94ce9e267b29e89
Sep 09 19:57:04.391560 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.391334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9f8fcccd-g4x64"]
Sep 09 19:57:04.391560 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.391430    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9"]
Sep 09 19:57:04.394551 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:57:04.394499    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8f43e6_f5ad_4fd6_922b_87611e4a0a32.slice/crio-791671b73e695be1cd75c536d8fe1452ff0818632940870b1932d0533f3be828 WatchSource:0}: Error finding container 791671b73e695be1cd75c536d8fe1452ff0818632940870b1932d0533f3be828: Status 404 returned error can't find the container with id 791671b73e695be1cd75c536d8fe1452ff0818632940870b1932d0533f3be828
Sep 09 19:57:04.420716 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.420683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4"]
Sep 09 19:57:04.423638 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:57:04.423607    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8f3e40_bf9e_415d_86b9_9142516823e3.slice/crio-1732ba659894628351d38a2e9a4d31b959b1525774c720e036708fd11ccd682e WatchSource:0}: Error finding container 1732ba659894628351d38a2e9a4d31b959b1525774c720e036708fd11ccd682e: Status 404 returned error can't find the container with id 1732ba659894628351d38a2e9a4d31b959b1525774c720e036708fd11ccd682e
Sep 09 19:57:04.428907 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.428774    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5475f4dd97-ldk2k"]
Sep 09 19:57:04.431482 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:57:04.431428    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded369a29_24d0_4290_ad74_c7b9cab47ba1.slice/crio-22c089dbb0ea0afd251b6ce50a7367c86f4708c7120b85f2497653a463d43066 WatchSource:0}: Error finding container 22c089dbb0ea0afd251b6ce50a7367c86f4708c7120b85f2497653a463d43066: Status 404 returned error can't find the container with id 22c089dbb0ea0afd251b6ce50a7367c86f4708c7120b85f2497653a463d43066
Sep 09 19:57:04.435191 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.435166    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:57:04.441431 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:57:04.439838    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365dd335_87a8_4c08_a25a_74db86c680ff.slice/crio-fe1befec094746827575c3125fd1fc72d4150657f9649b8209c1f608b4de82f2 WatchSource:0}: Error finding container fe1befec094746827575c3125fd1fc72d4150657f9649b8209c1f608b4de82f2: Status 404 returned error can't find the container with id fe1befec094746827575c3125fd1fc72d4150657f9649b8209c1f608b4de82f2
Sep 09 19:57:04.928090 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.928049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4" event={"ID":"8f8f3e40-bf9e-415d-86b9-9142516823e3","Type":"ContainerStarted","Data":"0b9acfb82ba3df9e66a222ef52930d279b1ebed767b13ad506b00167ffac83a1"}
Sep 09 19:57:04.928281 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.928161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4" event={"ID":"8f8f3e40-bf9e-415d-86b9-9142516823e3","Type":"ContainerStarted","Data":"9d014bc41eb0036fb02c4a4dc1fcf7261a2f8816241fbab27ab45ab0cee89111"}
Sep 09 19:57:04.928281 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.928197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4" event={"ID":"8f8f3e40-bf9e-415d-86b9-9142516823e3","Type":"ContainerStarted","Data":"1732ba659894628351d38a2e9a4d31b959b1525774c720e036708fd11ccd682e"}
Sep 09 19:57:04.929669 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.929629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"fe1befec094746827575c3125fd1fc72d4150657f9649b8209c1f608b4de82f2"}
Sep 09 19:57:04.931289 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.931255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9" event={"ID":"b2884be2-bbc7-4511-a78d-9933573cee86","Type":"ContainerStarted","Data":"4ef42a9d9e16fce781b9baa05e15fac9d8971a8dd034d712ab095360f4d4b3f8"}
Sep 09 19:57:04.933067 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.933037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-ctbm8" event={"ID":"85dabe45-811b-4f26-80e0-e6c00f8a8048","Type":"ContainerStarted","Data":"ddc943b4c3c0573a57f8a5dc1abeb1c831d6029aba9206c5c140f6fd60053774"}
Sep 09 19:57:04.933582 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.933549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-ctbm8"
Sep 09 19:57:04.934945 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.934909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k" event={"ID":"ed369a29-24d0-4290-ad74-c7b9cab47ba1","Type":"ContainerStarted","Data":"22c089dbb0ea0afd251b6ce50a7367c86f4708c7120b85f2497653a463d43066"}
Sep 09 19:57:04.936520 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.936491    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f8fcccd-g4x64" event={"ID":"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32","Type":"ContainerStarted","Data":"791671b73e695be1cd75c536d8fe1452ff0818632940870b1932d0533f3be828"}
Sep 09 19:57:04.937878 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.937853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5mbwq" event={"ID":"99e876b6-dc17-4c45-bf5d-2ee473f6bf73","Type":"ContainerStarted","Data":"a9577af8834aad6e6365887ed07ff2bf8efd556bcc89b6e7d94ce9e267b29e89"}
Sep 09 19:57:04.949757 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.949725    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-ctbm8"
Sep 09 19:57:04.953212 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:04.953157    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-ctbm8" podStartSLOduration=2.288910081 podStartE2EDuration="19.953140097s" totalImagesPullingTime="17.664230016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:04.951609042 +0000 UTC m=+112.979140030" watchObservedRunningTime="2026-09-09 19:57:04.953140097 +0000 UTC m=+112.980671083"
Sep 09 19:57:05.237568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:05.237532    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:05.307051 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:05.306991    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:08.863962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:08.863909    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7668c4ccf7-tm9rq"]
Sep 09 19:57:08.870337 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:08.870206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:57:08.956426 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:08.956330    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:09.035656 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:09.035622    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:09.176979 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:09.176889    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:09.623815 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:09.623783    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:10.975575 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:10.975533    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4" event={"ID":"8f8f3e40-bf9e-415d-86b9-9142516823e3","Type":"ContainerStarted","Data":"3d50eab1497a288024582008327e765c654b7a7beba4c3237fcc0a74f9ef3f30"}
Sep 09 19:57:10.979405 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:10.979355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"}
Sep 09 19:57:10.982207 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:10.982150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k" event={"ID":"ed369a29-24d0-4290-ad74-c7b9cab47ba1","Type":"ContainerStarted","Data":"5f269f17c04fb7bc4245adc360e829b26981592543a774d433d218509c4a604e"}
Sep 09 19:57:10.986790 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:10.986717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5mbwq" event={"ID":"99e876b6-dc17-4c45-bf5d-2ee473f6bf73","Type":"ContainerStarted","Data":"fa99ebba908549e6c0d75c2c98bff730d5b83f2e2f1cbcc51e4a2fb6a19b12b0"}
Sep 09 19:57:11.035044 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:11.034994    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-d26g4" podStartSLOduration=8.957643706 podStartE2EDuration="15.034973779s" totalImagesPullingTime="6.077330073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:11.001387438 +0000 UTC m=+119.028918425" watchObservedRunningTime="2026-09-09 19:57:11.034973779 +0000 UTC m=+119.062504782"
Sep 09 19:57:11.079353 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:11.079159    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k" podStartSLOduration=4.855073655 podStartE2EDuration="11.079135448s" totalImagesPullingTime="6.224061793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:11.055039812 +0000 UTC m=+119.082570799" watchObservedRunningTime="2026-09-09 19:57:11.079135448 +0000 UTC m=+119.106666435"
Sep 09 19:57:11.582275 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:11.582243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:11.994783 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:11.994743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9" event={"ID":"b2884be2-bbc7-4511-a78d-9933573cee86","Type":"ContainerStarted","Data":"9c83ce9e1c0518a161f994fd57f084f650e59b1a6457bb5b194f2352c5067227"}
Sep 09 19:57:11.995323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:11.994811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9" event={"ID":"b2884be2-bbc7-4511-a78d-9933573cee86","Type":"ContainerStarted","Data":"958b3637ec7046060968287fdb475fa6dc9f8f42dc3c311ad6f88bc9dd0c8274"}
Sep 09 19:57:11.995323 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:11.994843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9" event={"ID":"b2884be2-bbc7-4511-a78d-9933573cee86","Type":"ContainerStarted","Data":"be47b0f006b8dd58fda2df04ce7031b696a3ea7d81e38d8b8e159d5bd9a075b6"}
Sep 09 19:57:11.997043 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:11.997011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f8fcccd-g4x64" event={"ID":"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32","Type":"ContainerStarted","Data":"addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b"}
Sep 09 19:57:12.005657 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.005618    2566 generic.go:356] "Generic (PLEG): container finished" podID="99e876b6-dc17-4c45-bf5d-2ee473f6bf73" containerID="fa99ebba908549e6c0d75c2c98bff730d5b83f2e2f1cbcc51e4a2fb6a19b12b0" exitCode=0
Sep 09 19:57:12.005833 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.005705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5mbwq" event={"ID":"99e876b6-dc17-4c45-bf5d-2ee473f6bf73","Type":"ContainerDied","Data":"fa99ebba908549e6c0d75c2c98bff730d5b83f2e2f1cbcc51e4a2fb6a19b12b0"}
Sep 09 19:57:12.005833 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.005820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5mbwq" event={"ID":"99e876b6-dc17-4c45-bf5d-2ee473f6bf73","Type":"ContainerStarted","Data":"05594b4e11dca41666565a26b355f8464de1f488b9980954b355ae241affbdc5"}
Sep 09 19:57:12.005949 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.005851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5mbwq" event={"ID":"99e876b6-dc17-4c45-bf5d-2ee473f6bf73","Type":"ContainerStarted","Data":"891ec941702b8f7800e274c1963a6afcb227698b8f060d7143eab0e3157998b7"}
Sep 09 19:57:12.007804 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.007779    2566 generic.go:356] "Generic (PLEG): container finished" podID="365dd335-87a8-4c08-a25a-74db86c680ff" containerID="8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15" exitCode=0
Sep 09 19:57:12.007943 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.007868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"}
Sep 09 19:57:12.019351 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.019300    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vxnm9" podStartSLOduration=9.445022768 podStartE2EDuration="16.019284322s" totalImagesPullingTime="6.574261554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:12.018733966 +0000 UTC m=+120.046264952" watchObservedRunningTime="2026-09-09 19:57:12.019284322 +0000 UTC m=+120.046815365"
Sep 09 19:57:12.022404 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.022349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:12.071670 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.071608    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5mbwq" podStartSLOduration=9.467044641 podStartE2EDuration="16.07159097s" totalImagesPullingTime="6.515392345s" totalInitContainerRuntime="89.153984ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:12.069469908 +0000 UTC m=+120.097000894" watchObservedRunningTime="2026-09-09 19:57:12.07159097 +0000 UTC m=+120.099121955"
Sep 09 19:57:12.089426 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.089338    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d9f8fcccd-g4x64" podStartSLOduration=4.818801191 podStartE2EDuration="11.089315946s" totalImagesPullingTime="6.270514755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:12.087926853 +0000 UTC m=+120.115457862" watchObservedRunningTime="2026-09-09 19:57:12.089315946 +0000 UTC m=+120.116846981"
Sep 09 19:57:12.131382 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:12.131323    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:15.023626 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:15.023590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"}
Sep 09 19:57:15.024030 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:15.023657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"}
Sep 09 19:57:15.024030 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:15.023682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"}
Sep 09 19:57:15.024030 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:15.023708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"}
Sep 09 19:57:15.024030 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:15.023729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"}
Sep 09 19:57:16.132135 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:16.132095    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:16.233476 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:16.233442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:17.035427 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:17.035392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerStarted","Data":"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"}
Sep 09 19:57:17.066099 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:17.066035    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.346031549 podStartE2EDuration="20.066015859s" totalImagesPullingTime="9.692448864s" totalInitContainerRuntime="27.535446ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:17.063521513 +0000 UTC m=+125.091052532" watchObservedRunningTime="2026-09-09 19:57:17.066015859 +0000 UTC m=+125.093546846"
Sep 09 19:57:21.029540 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:21.029500    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:21.159732 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:21.159690    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:21.159970 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:21.159772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:21.705240 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:21.705199    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:21.705533 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:21.705265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:21.710049 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:21.710028    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:22.061245 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.061217    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:57:22.230714 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.230672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:22.293458 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.293423    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:22.356577 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.356497    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:22.390833 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.390794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:57:22.393316 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.393281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2014487-2cbf-4ff3-a134-4a70e6b9a002-metrics-certs\") pod \"network-metrics-daemon-s6f8r\" (UID: \"b2014487-2cbf-4ff3-a134-4a70e6b9a002\") " pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:57:22.631143 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.631066    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvgdp\""
Sep 09 19:57:22.638671 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.638651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6f8r"
Sep 09 19:57:22.942683 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:22.942604    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s6f8r"]
Sep 09 19:57:22.945908 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:57:22.945881    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2014487_2cbf_4ff3_a134_4a70e6b9a002.slice/crio-f0e75650dfa2cba998a60d7b014dbdfcaff4bf5b216442ae29c00b78b43ba346 WatchSource:0}: Error finding container f0e75650dfa2cba998a60d7b014dbdfcaff4bf5b216442ae29c00b78b43ba346: Status 404 returned error can't find the container with id f0e75650dfa2cba998a60d7b014dbdfcaff4bf5b216442ae29c00b78b43ba346
Sep 09 19:57:23.063063 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:23.063028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s6f8r" event={"ID":"b2014487-2cbf-4ff3-a134-4a70e6b9a002","Type":"ContainerStarted","Data":"f0e75650dfa2cba998a60d7b014dbdfcaff4bf5b216442ae29c00b78b43ba346"}
Sep 09 19:57:25.076811 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:25.076777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s6f8r" event={"ID":"b2014487-2cbf-4ff3-a134-4a70e6b9a002","Type":"ContainerStarted","Data":"7fe83b9f69be6361e0010c8212b10066cf4dedda0a145485edc3625dc6ef68f8"}
Sep 09 19:57:25.077234 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:25.076829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s6f8r" event={"ID":"b2014487-2cbf-4ff3-a134-4a70e6b9a002","Type":"ContainerStarted","Data":"bebef4eebb7c2393c6d07f3eb23e01f7b053e7c94aa6ce6fca51ee2b5fbe4467"}
Sep 09 19:57:25.095851 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:25.095794    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s6f8r" podStartSLOduration=130.355769916 podStartE2EDuration="2m12.095776084s" totalImagesPullingTime="1.740006168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:25.093576692 +0000 UTC m=+133.121107702" watchObservedRunningTime="2026-09-09 19:57:25.095776084 +0000 UTC m=+133.123307111"
Sep 09 19:57:25.191825 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:25.191791    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:25.293556 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:25.293466    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:29.123579 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:29.123537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:29.182577 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:29.182535    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:29.582541 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:29.582502    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:29.637068 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:29.637031    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:33.891528 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:33.891480    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq" podUID="e9b2b862-97f2-428c-94d0-910711df0eb2" containerName="registry" containerID="cri-o://1d078d7c1bfd90a8a36c60c4e4f1ebdb19fc6aa4530987cf3db85b63f5ac6c0f" gracePeriod=30
Sep 09 19:57:34.112956 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.112924    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9b2b862-97f2-428c-94d0-910711df0eb2" containerID="1d078d7c1bfd90a8a36c60c4e4f1ebdb19fc6aa4530987cf3db85b63f5ac6c0f" exitCode=0
Sep 09 19:57:34.113114 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.113001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq" event={"ID":"e9b2b862-97f2-428c-94d0-910711df0eb2","Type":"ContainerDied","Data":"1d078d7c1bfd90a8a36c60c4e4f1ebdb19fc6aa4530987cf3db85b63f5ac6c0f"}
Sep 09 19:57:34.137019 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.136995    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:57:34.292148 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-bound-sa-token\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292148 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292164    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292198    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bz2p\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-kube-api-access-9bz2p\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-installation-pull-secrets\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292259    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-certificates\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-image-registry-private-configuration\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9b2b862-97f2-428c-94d0-910711df0eb2-ca-trust-extracted\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292443 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292404    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-trusted-ca\") pod \"e9b2b862-97f2-428c-94d0-910711df0eb2\" (UID: \"e9b2b862-97f2-428c-94d0-910711df0eb2\") "
Sep 09 19:57:34.292930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:34.292930 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.292934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:34.294811 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.294779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:34.295120 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.295087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-kube-api-access-9bz2p" (OuterVolumeSpecName: "kube-api-access-9bz2p") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "kube-api-access-9bz2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:34.295239 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.295142    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:34.295304 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.295252    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:34.295391 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.295339    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:34.301498 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.301469    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b2b862-97f2-428c-94d0-910711df0eb2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e9b2b862-97f2-428c-94d0-910711df0eb2" (UID: "e9b2b862-97f2-428c-94d0-910711df0eb2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:57:34.393707 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393669    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9b2b862-97f2-428c-94d0-910711df0eb2-ca-trust-extracted\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:34.393707 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393699    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-trusted-ca\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:34.393707 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393708    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-bound-sa-token\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:34.393707 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393716    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:34.393969 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393724    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bz2p\" (UniqueName: \"kubernetes.io/projected/e9b2b862-97f2-428c-94d0-910711df0eb2-kube-api-access-9bz2p\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:34.393969 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393733    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-installation-pull-secrets\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:34.393969 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393741    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9b2b862-97f2-428c-94d0-910711df0eb2-registry-certificates\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:34.393969 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:34.393750    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9b2b862-97f2-428c-94d0-910711df0eb2-image-registry-private-configuration\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:57:35.118499 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:35.118460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq" event={"ID":"e9b2b862-97f2-428c-94d0-910711df0eb2","Type":"ContainerDied","Data":"5bb99fd73c5820c6d44a0b66dbae7b7be194308443442b2c0d2db29fba00823b"}
Sep 09 19:57:35.118499 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:35.118486    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7668c4ccf7-tm9rq"
Sep 09 19:57:35.118962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:35.118534    2566 scope.go:117] "RemoveContainer" containerID="1d078d7c1bfd90a8a36c60c4e4f1ebdb19fc6aa4530987cf3db85b63f5ac6c0f"
Sep 09 19:57:35.137821 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:35.137792    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7668c4ccf7-tm9rq"]
Sep 09 19:57:35.141295 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:35.141271    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7668c4ccf7-tm9rq"]
Sep 09 19:57:36.469465 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:36.469430    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9b2b862-97f2-428c-94d0-910711df0eb2" path="/var/lib/kubelet/pods/e9b2b862-97f2-428c-94d0-910711df0eb2/volumes"
Sep 09 19:57:37.150155 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:37.150122    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:37.207004 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:37.206965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:39.285564 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:39.285527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:39.386280 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:39.386237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:41.164641 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:41.164611    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:41.168467 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:41.168443    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5475f4dd97-ldk2k"
Sep 09 19:57:44.587427 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:44.587345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:44.644955 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:57:44.644926    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:15.871534 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:15.871495    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:16.001023 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.000991    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:16.905733 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.905693    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:58:16.906208 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.906102    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="alertmanager" containerID="cri-o://4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8" gracePeriod=120
Sep 09 19:58:16.906208 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.906140    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-metric" containerID="cri-o://951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d" gracePeriod=120
Sep 09 19:58:16.906208 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.906171    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="prom-label-proxy" containerID="cri-o://b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640" gracePeriod=120
Sep 09 19:58:16.906394 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.906189    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy" containerID="cri-o://d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd" gracePeriod=120
Sep 09 19:58:16.906394 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.906141    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-web" containerID="cri-o://cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477" gracePeriod=120
Sep 09 19:58:16.906394 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:16.906149    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="config-reloader" containerID="cri-o://343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749" gracePeriod=120
Sep 09 19:58:17.291394 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291344    2566 generic.go:356] "Generic (PLEG): container finished" podID="365dd335-87a8-4c08-a25a-74db86c680ff" containerID="b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640" exitCode=0
Sep 09 19:58:17.291394 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291388    2566 generic.go:356] "Generic (PLEG): container finished" podID="365dd335-87a8-4c08-a25a-74db86c680ff" containerID="d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd" exitCode=0
Sep 09 19:58:17.291597 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291400    2566 generic.go:356] "Generic (PLEG): container finished" podID="365dd335-87a8-4c08-a25a-74db86c680ff" containerID="343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749" exitCode=0
Sep 09 19:58:17.291597 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291410    2566 generic.go:356] "Generic (PLEG): container finished" podID="365dd335-87a8-4c08-a25a-74db86c680ff" containerID="4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8" exitCode=0
Sep 09 19:58:17.291597 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"}
Sep 09 19:58:17.291597 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"}
Sep 09 19:58:17.291597 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"}
Sep 09 19:58:17.291597 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:17.291539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"}
Sep 09 19:58:18.151076 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.151047    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.257518 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257472    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-cluster-tls-config\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.257518 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.257780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-config-volume\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.257780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257634    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-tls-assets\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.257780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257659    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-metrics-client-ca\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.257780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257684    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.257780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257705    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fhmf\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-kube-api-access-5fhmf\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.257780 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257774    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.258058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257814    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-main-db\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.258058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257848    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.258058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257875    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-config-out\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.258058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257903    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-web-config\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.258058 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.257960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"365dd335-87a8-4c08-a25a-74db86c680ff\" (UID: \"365dd335-87a8-4c08-a25a-74db86c680ff\") "
Sep 09 19:58:18.258490 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.258079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:18.258490 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.258283    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-metrics-client-ca\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.258963 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.258931    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:18.259285 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.259260    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:58:18.260838 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.260807    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:58:18.261451 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.261230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:18.261451 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.261328    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:18.261451 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.261355    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:18.261651 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.261459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-config-volume" (OuterVolumeSpecName: "config-volume") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:18.261651 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.261515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:18.261840 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.261814    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-config-out" (OuterVolumeSpecName: "config-out") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:58:18.262796 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.262764    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-kube-api-access-5fhmf" (OuterVolumeSpecName: "kube-api-access-5fhmf") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "kube-api-access-5fhmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:58:18.266649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.266494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:18.273693 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.273666    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-web-config" (OuterVolumeSpecName: "web-config") pod "365dd335-87a8-4c08-a25a-74db86c680ff" (UID: "365dd335-87a8-4c08-a25a-74db86c680ff"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:18.298426 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.298393    2566 generic.go:356] "Generic (PLEG): container finished" podID="365dd335-87a8-4c08-a25a-74db86c680ff" containerID="951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d" exitCode=0
Sep 09 19:58:18.298426 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.298418    2566 generic.go:356] "Generic (PLEG): container finished" podID="365dd335-87a8-4c08-a25a-74db86c680ff" containerID="cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477" exitCode=0
Sep 09 19:58:18.298674 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.298481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.298674 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.298476    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"}
Sep 09 19:58:18.298674 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.298607    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"}
Sep 09 19:58:18.298674 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.298657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"365dd335-87a8-4c08-a25a-74db86c680ff","Type":"ContainerDied","Data":"fe1befec094746827575c3125fd1fc72d4150657f9649b8209c1f608b4de82f2"}
Sep 09 19:58:18.298674 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.298668    2566 scope.go:117] "RemoveContainer" containerID="b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"
Sep 09 19:58:18.307251 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.307232    2566 scope.go:117] "RemoveContainer" containerID="951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"
Sep 09 19:58:18.314976 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.314956    2566 scope.go:117] "RemoveContainer" containerID="d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"
Sep 09 19:58:18.323480 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.323441    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:58:18.325159 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.325133    2566 scope.go:117] "RemoveContainer" containerID="cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"
Sep 09 19:58:18.331634 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.331613    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:58:18.336890 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.336871    2566 scope.go:117] "RemoveContainer" containerID="343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"
Sep 09 19:58:18.344659 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.344619    2566 scope.go:117] "RemoveContainer" containerID="4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"
Sep 09 19:58:18.352339 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.352302    2566 scope.go:117] "RemoveContainer" containerID="8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"
Sep 09 19:58:18.354660 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354640    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:58:18.354912 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354901    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="alertmanager"
Sep 09 19:58:18.354948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354914    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="alertmanager"
Sep 09 19:58:18.354948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354924    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="prom-label-proxy"
Sep 09 19:58:18.354948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354930    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="prom-label-proxy"
Sep 09 19:58:18.354948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354940    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy"
Sep 09 19:58:18.354948 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354945    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354953    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-web"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354959    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-web"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354969    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="init-config-reloader"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354975    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="init-config-reloader"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354984    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b2b862-97f2-428c-94d0-910711df0eb2" containerName="registry"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354989    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b2b862-97f2-428c-94d0-910711df0eb2" containerName="registry"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.354997    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-metric"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355002    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-metric"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355008    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="config-reloader"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355013    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="config-reloader"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355061    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-metric"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355069    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b2b862-97f2-428c-94d0-910711df0eb2" containerName="registry"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355076    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="alertmanager"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355080    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="prom-label-proxy"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355087    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="config-reloader"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355092    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy-web"
Sep 09 19:58:18.355166 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.355096    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" containerName="kube-rbac-proxy"
Sep 09 19:58:18.359041 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359019    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-cluster-tls-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359045    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359056    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-config-volume\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359065    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-tls-assets\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359074    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-main-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359086    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fhmf\" (UniqueName: \"kubernetes.io/projected/365dd335-87a8-4c08-a25a-74db86c680ff-kube-api-access-5fhmf\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359094    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359102    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-main-db\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359110    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365dd335-87a8-4c08-a25a-74db86c680ff-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359118    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/365dd335-87a8-4c08-a25a-74db86c680ff-config-out\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359124 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359125    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-web-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.359439 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.359133    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/365dd335-87a8-4c08-a25a-74db86c680ff-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:58:18.360155 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.360135    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.360480 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.360381    2566 scope.go:117] "RemoveContainer" containerID="b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"
Sep 09 19:58:18.360766 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:58:18.360739    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640\": container with ID starting with b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640 not found: ID does not exist" containerID="b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"
Sep 09 19:58:18.360834 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.360778    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"} err="failed to get container status \"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640\": rpc error: code = NotFound desc = could not find container \"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640\": container with ID starting with b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640 not found: ID does not exist"
Sep 09 19:58:18.360834 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.360822    2566 scope.go:117] "RemoveContainer" containerID="951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"
Sep 09 19:58:18.361107 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:58:18.361086    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d\": container with ID starting with 951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d not found: ID does not exist" containerID="951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"
Sep 09 19:58:18.361162 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.361113    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"} err="failed to get container status \"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d\": rpc error: code = NotFound desc = could not find container \"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d\": container with ID starting with 951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d not found: ID does not exist"
Sep 09 19:58:18.361162 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.361135    2566 scope.go:117] "RemoveContainer" containerID="d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"
Sep 09 19:58:18.361379 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:58:18.361341    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd\": container with ID starting with d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd not found: ID does not exist" containerID="d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"
Sep 09 19:58:18.361500 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.361398    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"} err="failed to get container status \"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd\": rpc error: code = NotFound desc = could not find container \"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd\": container with ID starting with d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd not found: ID does not exist"
Sep 09 19:58:18.361500 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.361422    2566 scope.go:117] "RemoveContainer" containerID="cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"
Sep 09 19:58:18.361755 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:58:18.361727    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477\": container with ID starting with cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477 not found: ID does not exist" containerID="cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"
Sep 09 19:58:18.361824 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.361757    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"} err="failed to get container status \"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477\": rpc error: code = NotFound desc = could not find container \"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477\": container with ID starting with cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477 not found: ID does not exist"
Sep 09 19:58:18.361824 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.361774    2566 scope.go:117] "RemoveContainer" containerID="343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"
Sep 09 19:58:18.362017 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:58:18.362002    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749\": container with ID starting with 343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749 not found: ID does not exist" containerID="343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"
Sep 09 19:58:18.362076 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362019    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"} err="failed to get container status \"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749\": rpc error: code = NotFound desc = could not find container \"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749\": container with ID starting with 343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749 not found: ID does not exist"
Sep 09 19:58:18.362076 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362033    2566 scope.go:117] "RemoveContainer" containerID="4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"
Sep 09 19:58:18.362304 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:58:18.362285    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8\": container with ID starting with 4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8 not found: ID does not exist" containerID="4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"
Sep 09 19:58:18.362398 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362312    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"} err="failed to get container status \"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8\": rpc error: code = NotFound desc = could not find container \"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8\": container with ID starting with 4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8 not found: ID does not exist"
Sep 09 19:58:18.362398 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362330    2566 scope.go:117] "RemoveContainer" containerID="8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"
Sep 09 19:58:18.362594 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:58:18.362578    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15\": container with ID starting with 8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15 not found: ID does not exist" containerID="8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"
Sep 09 19:58:18.362649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362598    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"} err="failed to get container status \"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15\": rpc error: code = NotFound desc = could not find container \"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15\": container with ID starting with 8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15 not found: ID does not exist"
Sep 09 19:58:18.362649 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362613    2566 scope.go:117] "RemoveContainer" containerID="b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"
Sep 09 19:58:18.362890 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362872    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 19:58:18.362998 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362871    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640"} err="failed to get container status \"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640\": rpc error: code = NotFound desc = could not find container \"b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640\": container with ID starting with b26613944b75da196f270772368f6eef0a2d6e3e0480c6e0ccf9ab450e939640 not found: ID does not exist"
Sep 09 19:58:18.362998 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.362966    2566 scope.go:117] "RemoveContainer" containerID="951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"
Sep 09 19:58:18.363140 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 19:58:18.363140 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 19:58:18.363271 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 19:58:18.363271 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363240    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 19:58:18.363406 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363296    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 19:58:18.363406 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363301    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d"} err="failed to get container status \"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d\": rpc error: code = NotFound desc = could not find container \"951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d\": container with ID starting with 951b8ea25ccc623a84a1a39d33175055116275e0c64908f6c2b129b535ad610d not found: ID does not exist"
Sep 09 19:58:18.363406 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363321    2566 scope.go:117] "RemoveContainer" containerID="d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"
Sep 09 19:58:18.363588 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363472    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7cc46\""
Sep 09 19:58:18.363588 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 19:58:18.363687 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 19:58:18.363687 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363659    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd"} err="failed to get container status \"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd\": rpc error: code = NotFound desc = could not find container \"d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd\": container with ID starting with d89f445849d9173bae4b47c6aedb4a09ee455c6c5c31f3e20d567f445dabb8cd not found: ID does not exist"
Sep 09 19:58:18.363687 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363680    2566 scope.go:117] "RemoveContainer" containerID="cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"
Sep 09 19:58:18.363971 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363950    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477"} err="failed to get container status \"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477\": rpc error: code = NotFound desc = could not find container \"cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477\": container with ID starting with cb98b42a78a40afa8b207f4450c09654541b05a24966968150928e8b30214477 not found: ID does not exist"
Sep 09 19:58:18.364021 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.363972    2566 scope.go:117] "RemoveContainer" containerID="343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"
Sep 09 19:58:18.364251 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.364233    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749"} err="failed to get container status \"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749\": rpc error: code = NotFound desc = could not find container \"343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749\": container with ID starting with 343202ff085e715b5be262c686705639868db0535e92789e8d7e5d0377c82749 not found: ID does not exist"
Sep 09 19:58:18.364314 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.364253    2566 scope.go:117] "RemoveContainer" containerID="4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"
Sep 09 19:58:18.364571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.364549    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8"} err="failed to get container status \"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8\": rpc error: code = NotFound desc = could not find container \"4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8\": container with ID starting with 4dce451abca9ff946ff88741d2c69d1ee14fa2b7c66162dc44f34da2d0111fb8 not found: ID does not exist"
Sep 09 19:58:18.364636 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.364573    2566 scope.go:117] "RemoveContainer" containerID="8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"
Sep 09 19:58:18.364892 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.364869    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15"} err="failed to get container status \"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15\": rpc error: code = NotFound desc = could not find container \"8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15\": container with ID starting with 8eb43eae6f33459b79f61c2cfae647edd5dc91b29854e2a3bccb7b7c94e64a15 not found: ID does not exist"
Sep 09 19:58:18.368313 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.368292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 19:58:18.376932 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.376910    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:58:18.459907 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.459868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.459907 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.459910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460111 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.459934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-web-config\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460111 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-config-out\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460111 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460111 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460111 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460111 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460287 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2db9r\" (UniqueName: \"kubernetes.io/projected/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-kube-api-access-2db9r\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460287 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460287 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460287 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.460438 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.460286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.468737 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.468710    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="365dd335-87a8-4c08-a25a-74db86c680ff" path="/var/lib/kubelet/pods/365dd335-87a8-4c08-a25a-74db86c680ff/volumes"
Sep 09 19:58:18.561356 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2db9r\" (UniqueName: \"kubernetes.io/projected/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-kube-api-access-2db9r\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.561356 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.561567 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.561567 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.561567 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.561567 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-web-config\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-config-out\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.561877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.562283 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.562186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.564076 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.564042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.564643 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.564614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.564804 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.564779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.564951 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.564927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-config-out\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.565044 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.564907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.565101 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.565076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.565429 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.565412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.565522 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.565428    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-web-config\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.565715 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.565695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.566481 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.566462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.566815 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.566799    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.570393 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.570351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2db9r\" (UniqueName: \"kubernetes.io/projected/c4945bfb-c5d8-4b61-93e1-e4f69e7879d6-kube-api-access-2db9r\") pod \"alertmanager-main-0\" (UID: \"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.670120 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.670076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:58:18.954571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:18.954332    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:58:18.957261 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:58:18.957229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4945bfb_c5d8_4b61_93e1_e4f69e7879d6.slice/crio-b9a747d0af6f9d4a2b5f8e53d20f7adeaa68da09525242687779ecbd948db6a3 WatchSource:0}: Error finding container b9a747d0af6f9d4a2b5f8e53d20f7adeaa68da09525242687779ecbd948db6a3: Status 404 returned error can't find the container with id b9a747d0af6f9d4a2b5f8e53d20f7adeaa68da09525242687779ecbd948db6a3
Sep 09 19:58:19.304345 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:19.304309    2566 generic.go:356] "Generic (PLEG): container finished" podID="c4945bfb-c5d8-4b61-93e1-e4f69e7879d6" containerID="771a28b6523564b905f134f2e80c1f09958e2719cad24c9a23b8c752486d53d5" exitCode=0
Sep 09 19:58:19.304747 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:19.304400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerDied","Data":"771a28b6523564b905f134f2e80c1f09958e2719cad24c9a23b8c752486d53d5"}
Sep 09 19:58:19.304747 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:19.304472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerStarted","Data":"b9a747d0af6f9d4a2b5f8e53d20f7adeaa68da09525242687779ecbd948db6a3"}
Sep 09 19:58:20.311817 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.311781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerStarted","Data":"311acf6e76c124a9fa352ecdab14c7ad4568ea20acc4c43856d636184dd00619"}
Sep 09 19:58:20.312284 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.311875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerStarted","Data":"f19c40cdf4bafaf1264df20030d02cb46c833d4e08e6f21256d3a7fd24f2f63b"}
Sep 09 19:58:20.312284 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.311915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerStarted","Data":"b1bcaf8d7b78bb18193c3568e3b5eebab896bc8e37e9a3b575ac6a864b0a0c9e"}
Sep 09 19:58:20.312284 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.311938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerStarted","Data":"4752ce8e3148239fcbda88675f829b01217d5984db4dcdf8afb845b95d166cae"}
Sep 09 19:58:20.312284 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.311963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerStarted","Data":"c05d84acee9b21d068b6229f65e0f3012dc50f8077e41a5ddbafd73e3d878326"}
Sep 09 19:58:20.312284 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.311987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c4945bfb-c5d8-4b61-93e1-e4f69e7879d6","Type":"ContainerStarted","Data":"dec6b189781e35263ccc25ebe22b08503d60d5521716468e9ea10a31fe4aa6ae"}
Sep 09 19:58:20.341551 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.341496    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.323424653 podStartE2EDuration="2.341477965s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.053312ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:20.338888723 +0000 UTC m=+188.366419720" watchObservedRunningTime="2026-09-09 19:58:20.341477965 +0000 UTC m=+188.369008952"
Sep 09 19:58:20.942016 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.941983    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"]
Sep 09 19:58:20.945521 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.945503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:20.948378 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.948339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 19:58:20.948568 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.948456    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 19:58:20.948729 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.948713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 19:58:20.948825 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.948745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d8rwb\""
Sep 09 19:58:20.948906 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.948837    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 19:58:20.949823 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.949802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 19:58:20.954056 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.954038    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 19:58:20.958806 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:20.958780    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"]
Sep 09 19:58:21.086151 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-secret-telemeter-client\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.086307 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-telemeter-client-tls\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.086307 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-metrics-client-ca\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.086307 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-federate-client-tls\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.086307 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9gf\" (UniqueName: \"kubernetes.io/projected/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-kube-api-access-qt9gf\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.086307 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-serving-certs-ca-bundle\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.086613 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.086613 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.086416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187308 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-metrics-client-ca\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187308 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-federate-client-tls\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9gf\" (UniqueName: \"kubernetes.io/projected/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-kube-api-access-qt9gf\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-serving-certs-ca-bundle\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-secret-telemeter-client\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.187571 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.187561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-telemeter-client-tls\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.188457 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.188424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-metrics-client-ca\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.188588 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.188435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-serving-certs-ca-bundle\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.188654 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.188593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.190217 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.190193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-telemeter-client-tls\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.190334 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.190303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-federate-client-tls\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.190614 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.190596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.190664 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.190645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-secret-telemeter-client\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.197423 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.197346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9gf\" (UniqueName: \"kubernetes.io/projected/7eb5bb4f-55bc-4d64-9348-0e82040b8fda-kube-api-access-qt9gf\") pod \"telemeter-client-78d5f9b79f-4qrw4\" (UID: \"7eb5bb4f-55bc-4d64-9348-0e82040b8fda\") " pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.256196 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.256158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"
Sep 09 19:58:21.541961 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:21.541932    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4"]
Sep 09 19:58:21.543006 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:58:21.542978    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb5bb4f_55bc_4d64_9348_0e82040b8fda.slice/crio-d15832e832e74e00b7bffd7a06a734f3c4ebcdbdeb3eaffdee7cbc356d2821d5 WatchSource:0}: Error finding container d15832e832e74e00b7bffd7a06a734f3c4ebcdbdeb3eaffdee7cbc356d2821d5: Status 404 returned error can't find the container with id d15832e832e74e00b7bffd7a06a734f3c4ebcdbdeb3eaffdee7cbc356d2821d5
Sep 09 19:58:22.327643 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:22.327604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4" event={"ID":"7eb5bb4f-55bc-4d64-9348-0e82040b8fda","Type":"ContainerStarted","Data":"d15832e832e74e00b7bffd7a06a734f3c4ebcdbdeb3eaffdee7cbc356d2821d5"}
Sep 09 19:58:23.334565 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:23.334526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4" event={"ID":"7eb5bb4f-55bc-4d64-9348-0e82040b8fda","Type":"ContainerStarted","Data":"1e83a019f191ef7b6ac6c7680e4a0b0252824e801fbfb99b51f9cddc1b54b70e"}
Sep 09 19:58:24.340959 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.340919    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4" event={"ID":"7eb5bb4f-55bc-4d64-9348-0e82040b8fda","Type":"ContainerStarted","Data":"8466de038f7453a88b21b7ccb9649103b35187df8a98dcc9c643f0c59cf47f39"}
Sep 09 19:58:24.341436 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.340987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4" event={"ID":"7eb5bb4f-55bc-4d64-9348-0e82040b8fda","Type":"ContainerStarted","Data":"6c46b114aa1209a41a05b8423bd52b838c7f6b47512794908e6adab147aee873"}
Sep 09 19:58:24.365684 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.365632    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78d5f9b79f-4qrw4" podStartSLOduration=2.676836025 podStartE2EDuration="4.365614249s" totalImagesPullingTime="1.688778224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:24.36430254 +0000 UTC m=+192.391833549" watchObservedRunningTime="2026-09-09 19:58:24.365614249 +0000 UTC m=+192.393145232"
Sep 09 19:58:24.459014 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.458980    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-859cbc67bc-wq4zw"]
Sep 09 19:58:24.462805 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.462782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.473688 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.473224    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-859cbc67bc-wq4zw"]
Sep 09 19:58:24.618744 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.618658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pc2\" (UniqueName: \"kubernetes.io/projected/0bdce081-f593-40d2-839e-e980b55ed936-kube-api-access-j9pc2\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.618744 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.618707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-service-ca\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.618744 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.618730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-serving-cert\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.618962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.618801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-oauth-config\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.618962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.618848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-console-config\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.618962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.618868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-oauth-serving-cert\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.618962 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.618896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-trusted-ca-bundle\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.719751 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.719704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pc2\" (UniqueName: \"kubernetes.io/projected/0bdce081-f593-40d2-839e-e980b55ed936-kube-api-access-j9pc2\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.719751 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.719763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-service-ca\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.719915 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.719790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-serving-cert\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.719915 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.719808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-oauth-config\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.719915 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.719828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-console-config\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.719915 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.719843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-oauth-serving-cert\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.719915 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.719880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-trusted-ca-bundle\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.720693 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.720665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-console-config\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.720803 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.720737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-trusted-ca-bundle\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.721025 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.721001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-service-ca\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.721068 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.721005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-oauth-serving-cert\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.722665 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.722636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:24.722784 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.722764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-oauth-config\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.722824 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.722764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-serving-cert\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.728110 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.728088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pc2\" (UniqueName: \"kubernetes.io/projected/0bdce081-f593-40d2-839e-e980b55ed936-kube-api-access-j9pc2\") pod \"console-859cbc67bc-wq4zw\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") " pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:24.774845 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:24.774800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:25.053439 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:25.053408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-859cbc67bc-wq4zw"]
Sep 09 19:58:25.056426 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:58:25.056353    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdce081_f593_40d2_839e_e980b55ed936.slice/crio-56a5bb10273bf413b0a4d3c4f2137ce33ff7a22e4996829d9342f534262eeb04 WatchSource:0}: Error finding container 56a5bb10273bf413b0a4d3c4f2137ce33ff7a22e4996829d9342f534262eeb04: Status 404 returned error can't find the container with id 56a5bb10273bf413b0a4d3c4f2137ce33ff7a22e4996829d9342f534262eeb04
Sep 09 19:58:25.347238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:25.347148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859cbc67bc-wq4zw" event={"ID":"0bdce081-f593-40d2-839e-e980b55ed936","Type":"ContainerStarted","Data":"949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339"}
Sep 09 19:58:25.347238 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:25.347209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859cbc67bc-wq4zw" event={"ID":"0bdce081-f593-40d2-839e-e980b55ed936","Type":"ContainerStarted","Data":"56a5bb10273bf413b0a4d3c4f2137ce33ff7a22e4996829d9342f534262eeb04"}
Sep 09 19:58:25.365353 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:25.365297    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-859cbc67bc-wq4zw" podStartSLOduration=1.3652781950000001 podStartE2EDuration="1.365278195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:25.364951086 +0000 UTC m=+193.392482073" watchObservedRunningTime="2026-09-09 19:58:25.365278195 +0000 UTC m=+193.392809182"
Sep 09 19:58:34.775944 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:34.775901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:34.776422 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:34.775987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:34.780852 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:34.780829    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:35.390667 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:35.390639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 19:58:35.447433 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:35.447401    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9f8fcccd-g4x64"]
Sep 09 19:58:35.549819 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:35.549769    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:35.651596 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:35.651486    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:35.724281 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:35.724241    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:52.055229 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:58:52.055183    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:00.469526 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.469468    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d9f8fcccd-g4x64" podUID="7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" containerName="console" containerID="cri-o://addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b" gracePeriod=15
Sep 09 19:59:00.706220 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.706187    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:59:00.794848 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.794812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-trusted-ca-bundle\") pod \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") "
Sep 09 19:59:00.795053 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.794885    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-oauth-serving-cert\") pod \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") "
Sep 09 19:59:00.795053 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.794914    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-service-ca\") pod \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") "
Sep 09 19:59:00.795053 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.794937    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-serving-cert\") pod \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") "
Sep 09 19:59:00.795053 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.794999    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9qf9\" (UniqueName: \"kubernetes.io/projected/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-kube-api-access-g9qf9\") pod \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") "
Sep 09 19:59:00.795053 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.795021    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-oauth-config\") pod \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") "
Sep 09 19:59:00.795293 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.795082    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-config\") pod \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\" (UID: \"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32\") "
Sep 09 19:59:00.795340 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.795301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" (UID: "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:59:00.795503 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.795480    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" (UID: "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:59:00.795544 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.795495    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" (UID: "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:59:00.795578 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.795543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-config" (OuterVolumeSpecName: "console-config") pod "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" (UID: "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:59:00.797591 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.797555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" (UID: "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:59:00.797720 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.797642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-kube-api-access-g9qf9" (OuterVolumeSpecName: "kube-api-access-g9qf9") pod "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" (UID: "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32"). InnerVolumeSpecName "kube-api-access-g9qf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:59:00.797720 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.797659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" (UID: "7c8f43e6-f5ad-4fd6-922b-87611e4a0a32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:59:00.895693 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.895654    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-trusted-ca-bundle\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:59:00.895693 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.895687    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-oauth-serving-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:59:00.895693 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.895697    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-service-ca\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:59:00.895693 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.895705    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-serving-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:59:00.895944 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.895713    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9qf9\" (UniqueName: \"kubernetes.io/projected/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-kube-api-access-g9qf9\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:59:00.895944 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.895722    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-oauth-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:59:00.895944 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:00.895730    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32-console-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 19:59:01.495102 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.495066    2566 generic.go:356] "Generic (PLEG): container finished" podID="7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" containerID="addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b" exitCode=0
Sep 09 19:59:01.495551 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.495113    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f8fcccd-g4x64"
Sep 09 19:59:01.495551 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.495126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f8fcccd-g4x64" event={"ID":"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32","Type":"ContainerDied","Data":"addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b"}
Sep 09 19:59:01.495551 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.495165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f8fcccd-g4x64" event={"ID":"7c8f43e6-f5ad-4fd6-922b-87611e4a0a32","Type":"ContainerDied","Data":"791671b73e695be1cd75c536d8fe1452ff0818632940870b1932d0533f3be828"}
Sep 09 19:59:01.495551 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.495184    2566 scope.go:117] "RemoveContainer" containerID="addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b"
Sep 09 19:59:01.504630 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.504609    2566 scope.go:117] "RemoveContainer" containerID="addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b"
Sep 09 19:59:01.504926 ip-10-0-135-134 kubenswrapper[2566]: E0909 19:59:01.504907    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b\": container with ID starting with addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b not found: ID does not exist" containerID="addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b"
Sep 09 19:59:01.504985 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.504935    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b"} err="failed to get container status \"addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b\": rpc error: code = NotFound desc = could not find container \"addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b\": container with ID starting with addb29d25206115e9d1b4e07b2fb5d0655a2bbe2a0cc43d9084afd43a2111a4b not found: ID does not exist"
Sep 09 19:59:01.517687 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.517648    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9f8fcccd-g4x64"]
Sep 09 19:59:01.521889 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:01.521862    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d9f8fcccd-g4x64"]
Sep 09 19:59:02.469294 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:02.469259    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" path="/var/lib/kubelet/pods/7c8f43e6-f5ad-4fd6-922b-87611e4a0a32/volumes"
Sep 09 19:59:18.727829 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:18.727757    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:31.118453 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:31.118418    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:31.195996 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:31.195959    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:36.225513 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.225398    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-799759b59f-vncgb"]
Sep 09 19:59:36.226062 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.225875    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" containerName="console"
Sep 09 19:59:36.226062 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.225893    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" containerName="console"
Sep 09 19:59:36.226062 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.225990    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8f43e6-f5ad-4fd6-922b-87611e4a0a32" containerName="console"
Sep 09 19:59:36.229273 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.229252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.241224 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.241201    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799759b59f-vncgb"]
Sep 09 19:59:36.283550 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.283510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:36.346855 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.346819    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:36.393217 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.393176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-trusted-ca-bundle\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.393429 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.393232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-config\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.393429 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.393325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-oauth-config\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.393429 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.393389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb94m\" (UniqueName: \"kubernetes.io/projected/b97719be-a0e9-42d0-a601-7a5e9d67976b-kube-api-access-zb94m\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.393599 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.393430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-service-ca\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.393599 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.393465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-oauth-serving-cert\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.393599 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.393495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-serving-cert\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.494328 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.494233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-service-ca\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.494328 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.494277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-oauth-serving-cert\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.494328 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.494296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-serving-cert\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.494621 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.494441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-trusted-ca-bundle\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.494621 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.494511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-config\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.494621 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.494571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-oauth-config\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.494621 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.494615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb94m\" (UniqueName: \"kubernetes.io/projected/b97719be-a0e9-42d0-a601-7a5e9d67976b-kube-api-access-zb94m\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.495177 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.495090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-service-ca\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.495177 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.495209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-oauth-serving-cert\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.495485 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.495287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-config\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.495485 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.495354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-trusted-ca-bundle\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.497173 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.497140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-oauth-config\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.497277 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.497199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-serving-cert\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.504062 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.504039    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb94m\" (UniqueName: \"kubernetes.io/projected/b97719be-a0e9-42d0-a601-7a5e9d67976b-kube-api-access-zb94m\") pod \"console-799759b59f-vncgb\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") " pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.539064 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.539020    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:36.817022 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:36.816933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799759b59f-vncgb"]
Sep 09 19:59:36.819789 ip-10-0-135-134 kubenswrapper[2566]: W0909 19:59:36.819760    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97719be_a0e9_42d0_a601_7a5e9d67976b.slice/crio-43d094bbaea3aee992e2c46e5aa4705b688cb5c798b08cf34d2c8ade6c5ed0fa WatchSource:0}: Error finding container 43d094bbaea3aee992e2c46e5aa4705b688cb5c798b08cf34d2c8ade6c5ed0fa: Status 404 returned error can't find the container with id 43d094bbaea3aee992e2c46e5aa4705b688cb5c798b08cf34d2c8ade6c5ed0fa
Sep 09 19:59:37.645349 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:37.645316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799759b59f-vncgb" event={"ID":"b97719be-a0e9-42d0-a601-7a5e9d67976b","Type":"ContainerStarted","Data":"02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5"}
Sep 09 19:59:37.645774 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:37.645381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799759b59f-vncgb" event={"ID":"b97719be-a0e9-42d0-a601-7a5e9d67976b","Type":"ContainerStarted","Data":"43d094bbaea3aee992e2c46e5aa4705b688cb5c798b08cf34d2c8ade6c5ed0fa"}
Sep 09 19:59:37.667764 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:37.667714    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-799759b59f-vncgb" podStartSLOduration=1.667697337 podStartE2EDuration="1.667697337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:59:37.665326341 +0000 UTC m=+265.692857326" watchObservedRunningTime="2026-09-09 19:59:37.667697337 +0000 UTC m=+265.695228322"
Sep 09 19:59:43.076154 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:43.076120    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:43.160700 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:43.160659    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:46.539892 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:46.539848    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:46.540300 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:46.539928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:46.544919 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:46.544892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:46.683642 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:46.683613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 19:59:46.734512 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:46.734479    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-859cbc67bc-wq4zw"]
Sep 09 19:59:46.841755 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:46.841632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:47.088084 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:47.088048    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:47.189216 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:47.189084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:59.595652 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:59.595620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:59.696899 ip-10-0-135-134 kubenswrapper[2566]: I0909 19:59:59.696867    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:00:04.739531 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:04.739494    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:00:11.762995 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:11.762948    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-859cbc67bc-wq4zw" podUID="0bdce081-f593-40d2-839e-e980b55ed936" containerName="console" containerID="cri-o://949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339" gracePeriod=15
Sep 09 20:00:12.012784 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.012758    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 20:00:12.096592 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.096497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-oauth-serving-cert\") pod \"0bdce081-f593-40d2-839e-e980b55ed936\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") "
Sep 09 20:00:12.096592 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.096554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-serving-cert\") pod \"0bdce081-f593-40d2-839e-e980b55ed936\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") "
Sep 09 20:00:12.096592 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.096575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-oauth-config\") pod \"0bdce081-f593-40d2-839e-e980b55ed936\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") "
Sep 09 20:00:12.096856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.096624    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-service-ca\") pod \"0bdce081-f593-40d2-839e-e980b55ed936\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") "
Sep 09 20:00:12.096856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.096672    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-console-config\") pod \"0bdce081-f593-40d2-839e-e980b55ed936\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") "
Sep 09 20:00:12.096856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.096696    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-trusted-ca-bundle\") pod \"0bdce081-f593-40d2-839e-e980b55ed936\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") "
Sep 09 20:00:12.096856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.096735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9pc2\" (UniqueName: \"kubernetes.io/projected/0bdce081-f593-40d2-839e-e980b55ed936-kube-api-access-j9pc2\") pod \"0bdce081-f593-40d2-839e-e980b55ed936\" (UID: \"0bdce081-f593-40d2-839e-e980b55ed936\") "
Sep 09 20:00:12.097063 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.097041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0bdce081-f593-40d2-839e-e980b55ed936" (UID: "0bdce081-f593-40d2-839e-e980b55ed936"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:00:12.097214 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.097175    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-console-config" (OuterVolumeSpecName: "console-config") pod "0bdce081-f593-40d2-839e-e980b55ed936" (UID: "0bdce081-f593-40d2-839e-e980b55ed936"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:00:12.097349 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.097179    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-service-ca" (OuterVolumeSpecName: "service-ca") pod "0bdce081-f593-40d2-839e-e980b55ed936" (UID: "0bdce081-f593-40d2-839e-e980b55ed936"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:00:12.097349 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.097174    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0bdce081-f593-40d2-839e-e980b55ed936" (UID: "0bdce081-f593-40d2-839e-e980b55ed936"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:00:12.099071 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.099041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0bdce081-f593-40d2-839e-e980b55ed936" (UID: "0bdce081-f593-40d2-839e-e980b55ed936"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:00:12.099161 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.099067    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0bdce081-f593-40d2-839e-e980b55ed936" (UID: "0bdce081-f593-40d2-839e-e980b55ed936"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:00:12.099161 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.099116    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdce081-f593-40d2-839e-e980b55ed936-kube-api-access-j9pc2" (OuterVolumeSpecName: "kube-api-access-j9pc2") pod "0bdce081-f593-40d2-839e-e980b55ed936" (UID: "0bdce081-f593-40d2-839e-e980b55ed936"). InnerVolumeSpecName "kube-api-access-j9pc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:00:12.198025 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.197979    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-serving-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:00:12.198025 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.198015    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0bdce081-f593-40d2-839e-e980b55ed936-console-oauth-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:00:12.198025 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.198029    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-service-ca\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:00:12.198266 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.198040    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-console-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:00:12.198266 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.198052    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-trusted-ca-bundle\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:00:12.198266 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.198063    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pc2\" (UniqueName: \"kubernetes.io/projected/0bdce081-f593-40d2-839e-e980b55ed936-kube-api-access-j9pc2\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:00:12.198266 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.198074    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0bdce081-f593-40d2-839e-e980b55ed936-oauth-serving-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:00:12.404151 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.404076    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:00:12.408549 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.408518    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:00:12.415392 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.415347    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:00:12.786838 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.786796    2566 generic.go:356] "Generic (PLEG): container finished" podID="0bdce081-f593-40d2-839e-e980b55ed936" containerID="949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339" exitCode=0
Sep 09 20:00:12.787267 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.786860    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859cbc67bc-wq4zw"
Sep 09 20:00:12.787267 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.786858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859cbc67bc-wq4zw" event={"ID":"0bdce081-f593-40d2-839e-e980b55ed936","Type":"ContainerDied","Data":"949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339"}
Sep 09 20:00:12.787267 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.786975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859cbc67bc-wq4zw" event={"ID":"0bdce081-f593-40d2-839e-e980b55ed936","Type":"ContainerDied","Data":"56a5bb10273bf413b0a4d3c4f2137ce33ff7a22e4996829d9342f534262eeb04"}
Sep 09 20:00:12.787267 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.786999    2566 scope.go:117] "RemoveContainer" containerID="949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339"
Sep 09 20:00:12.798236 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.798210    2566 scope.go:117] "RemoveContainer" containerID="949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339"
Sep 09 20:00:12.798616 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:00:12.798589    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339\": container with ID starting with 949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339 not found: ID does not exist" containerID="949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339"
Sep 09 20:00:12.798733 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.798627    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339"} err="failed to get container status \"949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339\": rpc error: code = NotFound desc = could not find container \"949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339\": container with ID starting with 949e9a258b7f60621097413d8292a2b4a003bbfb9997723f7aa0564f0976e339 not found: ID does not exist"
Sep 09 20:00:12.805509 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.805485    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-859cbc67bc-wq4zw"]
Sep 09 20:00:12.809758 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:12.809737    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-859cbc67bc-wq4zw"]
Sep 09 20:00:14.469242 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:00:14.469209    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bdce081-f593-40d2-839e-e980b55ed936" path="/var/lib/kubelet/pods/0bdce081-f593-40d2-839e-e980b55ed936/volumes"
Sep 09 20:01:50.541942 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:01:50.541893    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:53.835474 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:01:53.835420    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:05.595251 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.595219    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c7d65d5c7-kdl2n"]
Sep 09 20:02:05.595861 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.595604    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bdce081-f593-40d2-839e-e980b55ed936" containerName="console"
Sep 09 20:02:05.595861 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.595623    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bdce081-f593-40d2-839e-e980b55ed936" containerName="console"
Sep 09 20:02:05.595861 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.595709    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bdce081-f593-40d2-839e-e980b55ed936" containerName="console"
Sep 09 20:02:05.598739 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.598718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.610591 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.610562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7d65d5c7-kdl2n"]
Sep 09 20:02:05.725350 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.725309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-oauth-config\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.725551 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.725393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvng\" (UniqueName: \"kubernetes.io/projected/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-kube-api-access-4zvng\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.725551 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.725424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-oauth-serving-cert\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.725551 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.725448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-serving-cert\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.725551 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.725508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-trusted-ca-bundle\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.725551 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.725532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-config\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.725713 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.725586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-service-ca\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.826155 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.826119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvng\" (UniqueName: \"kubernetes.io/projected/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-kube-api-access-4zvng\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.826155 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.826162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-oauth-serving-cert\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.826477 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.826186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-serving-cert\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.826477 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.826208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-trusted-ca-bundle\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.826477 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.826229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-config\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.826477 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.826268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-service-ca\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.826477 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.826347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-oauth-config\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.827070 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.827038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-config\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.827193 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.827041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-oauth-serving-cert\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.827258 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.827217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-trusted-ca-bundle\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.827309 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.827287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-service-ca\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.828961 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.828923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-serving-cert\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.829040 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.828968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-console-oauth-config\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.835743 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.835725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvng\" (UniqueName: \"kubernetes.io/projected/2532c5d5-9d93-44bf-8d3c-c21e54bcca38-kube-api-access-4zvng\") pod \"console-c7d65d5c7-kdl2n\" (UID: \"2532c5d5-9d93-44bf-8d3c-c21e54bcca38\") " pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:05.910499 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:05.910403    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:06.184170 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:06.184145    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7d65d5c7-kdl2n"]
Sep 09 20:02:06.186823 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:02:06.186781    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2532c5d5_9d93_44bf_8d3c_c21e54bcca38.slice/crio-1cff61dfe83bbb5ee35f74a6be90fb1072d2b6470739d682efb5ac5ee1312e0e WatchSource:0}: Error finding container 1cff61dfe83bbb5ee35f74a6be90fb1072d2b6470739d682efb5ac5ee1312e0e: Status 404 returned error can't find the container with id 1cff61dfe83bbb5ee35f74a6be90fb1072d2b6470739d682efb5ac5ee1312e0e
Sep 09 20:02:06.230730 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:06.230704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7d65d5c7-kdl2n" event={"ID":"2532c5d5-9d93-44bf-8d3c-c21e54bcca38","Type":"ContainerStarted","Data":"1cff61dfe83bbb5ee35f74a6be90fb1072d2b6470739d682efb5ac5ee1312e0e"}
Sep 09 20:02:06.376421 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:06.376388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:07.236351 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:07.236308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7d65d5c7-kdl2n" event={"ID":"2532c5d5-9d93-44bf-8d3c-c21e54bcca38","Type":"ContainerStarted","Data":"93b6cf71e1425a8caeefddf585284f58a7f9246c09a87e7e87fe8f8c9a1d39fd"}
Sep 09 20:02:07.255025 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:07.254973    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c7d65d5c7-kdl2n" podStartSLOduration=2.254955239 podStartE2EDuration="2.254955239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:07.253979771 +0000 UTC m=+415.281510790" watchObservedRunningTime="2026-09-09 20:02:07.254955239 +0000 UTC m=+415.282486224"
Sep 09 20:02:15.911582 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:15.911534    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:15.912131 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:15.911627    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:15.916588 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:15.916562    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:16.276007 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:16.275978    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c7d65d5c7-kdl2n"
Sep 09 20:02:16.331762 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:16.331726    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799759b59f-vncgb"]
Sep 09 20:02:16.448702 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:16.448667    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:16.551230 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:16.551134    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:16.648773 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:16.648733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:17.626616 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:17.626582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:17.746437 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:17.746403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:19.031478 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.031443    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"]
Sep 09 20:02:19.035131 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.035110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.038250 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.038231    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:02:19.038401 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.038304    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:02:19.038641 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.038621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-rs2hb\""
Sep 09 20:02:19.038990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.038968    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:02:19.060176 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.060143    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"]
Sep 09 20:02:19.144288 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.144242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lg4\" (UniqueName: \"kubernetes.io/projected/e592bbfd-8c00-4496-86b3-b1bc4db0334f-kube-api-access-p8lg4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djfmz\" (UID: \"e592bbfd-8c00-4496-86b3-b1bc4db0334f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.144511 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.144310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e592bbfd-8c00-4496-86b3-b1bc4db0334f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djfmz\" (UID: \"e592bbfd-8c00-4496-86b3-b1bc4db0334f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.245603 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.245564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lg4\" (UniqueName: \"kubernetes.io/projected/e592bbfd-8c00-4496-86b3-b1bc4db0334f-kube-api-access-p8lg4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djfmz\" (UID: \"e592bbfd-8c00-4496-86b3-b1bc4db0334f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.245777 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.245621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e592bbfd-8c00-4496-86b3-b1bc4db0334f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djfmz\" (UID: \"e592bbfd-8c00-4496-86b3-b1bc4db0334f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.248067 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.248044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e592bbfd-8c00-4496-86b3-b1bc4db0334f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djfmz\" (UID: \"e592bbfd-8c00-4496-86b3-b1bc4db0334f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.258456 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.258426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lg4\" (UniqueName: \"kubernetes.io/projected/e592bbfd-8c00-4496-86b3-b1bc4db0334f-kube-api-access-p8lg4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djfmz\" (UID: \"e592bbfd-8c00-4496-86b3-b1bc4db0334f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.346529 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.346428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:19.625072 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.625032    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"]
Sep 09 20:02:19.628060 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:02:19.628030    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode592bbfd_8c00_4496_86b3_b1bc4db0334f.slice/crio-d42db35936f13b364350d54238d71e15145459fe39345fd8126414bcd890bbeb WatchSource:0}: Error finding container d42db35936f13b364350d54238d71e15145459fe39345fd8126414bcd890bbeb: Status 404 returned error can't find the container with id d42db35936f13b364350d54238d71e15145459fe39345fd8126414bcd890bbeb
Sep 09 20:02:19.630197 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:19.630176    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:02:20.294379 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:20.294329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz" event={"ID":"e592bbfd-8c00-4496-86b3-b1bc4db0334f","Type":"ContainerStarted","Data":"d42db35936f13b364350d54238d71e15145459fe39345fd8126414bcd890bbeb"}
Sep 09 20:02:23.866376 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:23.866335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.117906 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.117814    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.124451 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.124419    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-w5sg7"]
Sep 09 20:02:24.127864 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.127843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.130562 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.130538    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:02:24.130688 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.130581    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-dmzh9\""
Sep 09 20:02:24.130688 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.130547    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:02:24.137806 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.137780    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-w5sg7"]
Sep 09 20:02:24.278680 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.278644    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.290681 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.290655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46bb560d-4267-4366-815e-a97fd9c5512c-certificates\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.290815 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.290704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/46bb560d-4267-4366-815e-a97fd9c5512c-cabundle0\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.290815 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.290727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf2wt\" (UniqueName: \"kubernetes.io/projected/46bb560d-4267-4366-815e-a97fd9c5512c-kube-api-access-qf2wt\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.320057 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.320020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz" event={"ID":"e592bbfd-8c00-4496-86b3-b1bc4db0334f","Type":"ContainerStarted","Data":"25ce801ac3ae241667a97314a15b60f596a87a08b32e2aebef593f42da2708ff"}
Sep 09 20:02:24.320245 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.320151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:24.340558 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.340500    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz" podStartSLOduration=1.565127376 podStartE2EDuration="5.340475472s" totalImagesPullingTime="3.775348096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:24.340004372 +0000 UTC m=+432.367535358" watchObservedRunningTime="2026-09-09 20:02:24.340475472 +0000 UTC m=+432.368006458"
Sep 09 20:02:24.353800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.353765    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.357580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.357548    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"]
Sep 09 20:02:24.361407 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.361386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.364091 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.364072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:02:24.371792 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.371732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"]
Sep 09 20:02:24.391307 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.391273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46bb560d-4267-4366-815e-a97fd9c5512c-certificates\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.391496 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.391325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/46bb560d-4267-4366-815e-a97fd9c5512c-cabundle0\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.391496 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.391346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf2wt\" (UniqueName: \"kubernetes.io/projected/46bb560d-4267-4366-815e-a97fd9c5512c-kube-api-access-qf2wt\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.392120 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.392096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/46bb560d-4267-4366-815e-a97fd9c5512c-cabundle0\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.394415 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.394391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46bb560d-4267-4366-815e-a97fd9c5512c-certificates\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.410918 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.410887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf2wt\" (UniqueName: \"kubernetes.io/projected/46bb560d-4267-4366-815e-a97fd9c5512c-kube-api-access-qf2wt\") pod \"keda-operator-579c8958f7-w5sg7\" (UID: \"46bb560d-4267-4366-815e-a97fd9c5512c\") " pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.439860 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.439826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:24.493809 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.492724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4px\" (UniqueName: \"kubernetes.io/projected/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-kube-api-access-mp4px\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.493809 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.492802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.493809 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.492870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.559098 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.559054    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.563255 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.563224    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ckf8f"]
Sep 09 20:02:24.566961 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.566938    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.569572 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.569544    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:02:24.577198 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.577161    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ckf8f"]
Sep 09 20:02:24.594308 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.594270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.594308 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.594313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.594582 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.594403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4px\" (UniqueName: \"kubernetes.io/projected/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-kube-api-access-mp4px\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.595076 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.594897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.598003 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.597976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.604197 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.604167    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4px\" (UniqueName: \"kubernetes.io/projected/f6ce128c-87ae-4722-b458-ec7e6d9cde1c-kube-api-access-mp4px\") pod \"keda-metrics-apiserver-77857b7b98-jlbnn\" (UID: \"f6ce128c-87ae-4722-b458-ec7e6d9cde1c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.667771 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.667671    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.672410 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.672386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:24.695112 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.695078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbr7t\" (UniqueName: \"kubernetes.io/projected/67493e5b-e438-44fe-9c4c-4e0b9cd4310c-kube-api-access-qbr7t\") pod \"keda-admission-6c4d864b86-ckf8f\" (UID: \"67493e5b-e438-44fe-9c4c-4e0b9cd4310c\") " pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.695254 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.695149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/67493e5b-e438-44fe-9c4c-4e0b9cd4310c-certificates\") pod \"keda-admission-6c4d864b86-ckf8f\" (UID: \"67493e5b-e438-44fe-9c4c-4e0b9cd4310c\") " pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.722241 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.722139    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-w5sg7"]
Sep 09 20:02:24.724771 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:02:24.724737    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bb560d_4267_4366_815e_a97fd9c5512c.slice/crio-286d57fbbb194263b4479dd8bb0a6f700ab59ff90d3d22bcaebe7dc6f9bdd435 WatchSource:0}: Error finding container 286d57fbbb194263b4479dd8bb0a6f700ab59ff90d3d22bcaebe7dc6f9bdd435: Status 404 returned error can't find the container with id 286d57fbbb194263b4479dd8bb0a6f700ab59ff90d3d22bcaebe7dc6f9bdd435
Sep 09 20:02:24.748911 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.748874    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.796514 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.796472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbr7t\" (UniqueName: \"kubernetes.io/projected/67493e5b-e438-44fe-9c4c-4e0b9cd4310c-kube-api-access-qbr7t\") pod \"keda-admission-6c4d864b86-ckf8f\" (UID: \"67493e5b-e438-44fe-9c4c-4e0b9cd4310c\") " pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.796684 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.796543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/67493e5b-e438-44fe-9c4c-4e0b9cd4310c-certificates\") pod \"keda-admission-6c4d864b86-ckf8f\" (UID: \"67493e5b-e438-44fe-9c4c-4e0b9cd4310c\") " pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.799351 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.799324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/67493e5b-e438-44fe-9c4c-4e0b9cd4310c-certificates\") pod \"keda-admission-6c4d864b86-ckf8f\" (UID: \"67493e5b-e438-44fe-9c4c-4e0b9cd4310c\") " pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.805401 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.805384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbr7t\" (UniqueName: \"kubernetes.io/projected/67493e5b-e438-44fe-9c4c-4e0b9cd4310c-kube-api-access-qbr7t\") pod \"keda-admission-6c4d864b86-ckf8f\" (UID: \"67493e5b-e438-44fe-9c4c-4e0b9cd4310c\") " pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.879412 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.879342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:24.907090 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.907036    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:24.956859 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:24.956545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"]
Sep 09 20:02:24.960068 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:02:24.960033    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ce128c_87ae_4722_b458_ec7e6d9cde1c.slice/crio-0ea02e9b9c3ca285d7359417d8ac43993e53ee05f3bb29d08c75b3319f978954 WatchSource:0}: Error finding container 0ea02e9b9c3ca285d7359417d8ac43993e53ee05f3bb29d08c75b3319f978954: Status 404 returned error can't find the container with id 0ea02e9b9c3ca285d7359417d8ac43993e53ee05f3bb29d08c75b3319f978954
Sep 09 20:02:25.119014 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.118963    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:25.161012 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.160930    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ckf8f"]
Sep 09 20:02:25.163137 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:02:25.163115    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67493e5b_e438_44fe_9c4c_4e0b9cd4310c.slice/crio-d69ed1a06ada1caeadb7410ba6bcbf7bfc90cecb45e2e4c2659d44ae4e8fd8b2 WatchSource:0}: Error finding container d69ed1a06ada1caeadb7410ba6bcbf7bfc90cecb45e2e4c2659d44ae4e8fd8b2: Status 404 returned error can't find the container with id d69ed1a06ada1caeadb7410ba6bcbf7bfc90cecb45e2e4c2659d44ae4e8fd8b2
Sep 09 20:02:25.273806 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.273775    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:25.326488 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.326451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn" event={"ID":"f6ce128c-87ae-4722-b458-ec7e6d9cde1c","Type":"ContainerStarted","Data":"0ea02e9b9c3ca285d7359417d8ac43993e53ee05f3bb29d08c75b3319f978954"}
Sep 09 20:02:25.327691 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.327659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-w5sg7" event={"ID":"46bb560d-4267-4366-815e-a97fd9c5512c","Type":"ContainerStarted","Data":"286d57fbbb194263b4479dd8bb0a6f700ab59ff90d3d22bcaebe7dc6f9bdd435"}
Sep 09 20:02:25.328856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.328830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ckf8f" event={"ID":"67493e5b-e438-44fe-9c4c-4e0b9cd4310c","Type":"ContainerStarted","Data":"d69ed1a06ada1caeadb7410ba6bcbf7bfc90cecb45e2e4c2659d44ae4e8fd8b2"}
Sep 09 20:02:25.379004 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.378973    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:25.477051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:25.477013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:29.353271 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.353183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn" event={"ID":"f6ce128c-87ae-4722-b458-ec7e6d9cde1c","Type":"ContainerStarted","Data":"76d55a7d01f2425b3e4099672596066165ded2c045211b479b9ac2f3572fac5a"}
Sep 09 20:02:29.353739 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.353331    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:29.355071 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.355031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-w5sg7" event={"ID":"46bb560d-4267-4366-815e-a97fd9c5512c","Type":"ContainerStarted","Data":"b44aeeb0fd0bfac779876fe2b6cdab1268a1ad527e7aca83ce57dead967a8293"}
Sep 09 20:02:29.355202 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.355112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:29.356760 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.356734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ckf8f" event={"ID":"67493e5b-e438-44fe-9c4c-4e0b9cd4310c","Type":"ContainerStarted","Data":"2afaeebdef5e44089220d1b97eed3d98b35d5d57a87ec2595dfcac1f7234ca00"}
Sep 09 20:02:29.356900 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.356880    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:02:29.372163 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.372107    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn" podStartSLOduration=1.287803319 podStartE2EDuration="5.372089802s" totalImagesPullingTime="4.084286483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:29.370033199 +0000 UTC m=+437.397564208" watchObservedRunningTime="2026-09-09 20:02:29.372089802 +0000 UTC m=+437.399620787"
Sep 09 20:02:29.386888 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.386839    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-ckf8f" podStartSLOduration=1.514258718 podStartE2EDuration="5.386824709s" totalImagesPullingTime="3.872565991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:29.3858915 +0000 UTC m=+437.413422497" watchObservedRunningTime="2026-09-09 20:02:29.386824709 +0000 UTC m=+437.414355695"
Sep 09 20:02:29.403653 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:29.403603    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-w5sg7" podStartSLOduration=1.082437708 podStartE2EDuration="5.403586396s" totalImagesPullingTime="4.321148688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:29.402545439 +0000 UTC m=+437.430076426" watchObservedRunningTime="2026-09-09 20:02:29.403586396 +0000 UTC m=+437.431117384"
Sep 09 20:02:40.365971 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:40.365933    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jlbnn"
Sep 09 20:02:41.353029 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.352978    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-799759b59f-vncgb" podUID="b97719be-a0e9-42d0-a601-7a5e9d67976b" containerName="console" containerID="cri-o://02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5" gracePeriod=15
Sep 09 20:02:41.609868 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.609799    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 20:02:41.656962 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.656923    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-serving-cert\") pod \"b97719be-a0e9-42d0-a601-7a5e9d67976b\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") "
Sep 09 20:02:41.657139 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.656986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-config\") pod \"b97719be-a0e9-42d0-a601-7a5e9d67976b\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") "
Sep 09 20:02:41.657139 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb94m\" (UniqueName: \"kubernetes.io/projected/b97719be-a0e9-42d0-a601-7a5e9d67976b-kube-api-access-zb94m\") pod \"b97719be-a0e9-42d0-a601-7a5e9d67976b\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") "
Sep 09 20:02:41.657139 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657097    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-trusted-ca-bundle\") pod \"b97719be-a0e9-42d0-a601-7a5e9d67976b\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") "
Sep 09 20:02:41.657139 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657118    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-service-ca\") pod \"b97719be-a0e9-42d0-a601-7a5e9d67976b\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") "
Sep 09 20:02:41.657340 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-oauth-serving-cert\") pod \"b97719be-a0e9-42d0-a601-7a5e9d67976b\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") "
Sep 09 20:02:41.657340 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-oauth-config\") pod \"b97719be-a0e9-42d0-a601-7a5e9d67976b\" (UID: \"b97719be-a0e9-42d0-a601-7a5e9d67976b\") "
Sep 09 20:02:41.657472 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657437    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-config" (OuterVolumeSpecName: "console-config") pod "b97719be-a0e9-42d0-a601-7a5e9d67976b" (UID: "b97719be-a0e9-42d0-a601-7a5e9d67976b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:41.657601 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657583    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:02:41.657681 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b97719be-a0e9-42d0-a601-7a5e9d67976b" (UID: "b97719be-a0e9-42d0-a601-7a5e9d67976b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:41.657947 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.657919    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-service-ca" (OuterVolumeSpecName: "service-ca") pod "b97719be-a0e9-42d0-a601-7a5e9d67976b" (UID: "b97719be-a0e9-42d0-a601-7a5e9d67976b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:41.658215 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.658004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b97719be-a0e9-42d0-a601-7a5e9d67976b" (UID: "b97719be-a0e9-42d0-a601-7a5e9d67976b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:41.659580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.659543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b97719be-a0e9-42d0-a601-7a5e9d67976b" (UID: "b97719be-a0e9-42d0-a601-7a5e9d67976b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:02:41.659683 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.659591    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b97719be-a0e9-42d0-a601-7a5e9d67976b" (UID: "b97719be-a0e9-42d0-a601-7a5e9d67976b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:02:41.659851 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.659827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97719be-a0e9-42d0-a601-7a5e9d67976b-kube-api-access-zb94m" (OuterVolumeSpecName: "kube-api-access-zb94m") pod "b97719be-a0e9-42d0-a601-7a5e9d67976b" (UID: "b97719be-a0e9-42d0-a601-7a5e9d67976b"). InnerVolumeSpecName "kube-api-access-zb94m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:02:41.758846 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.758812    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb94m\" (UniqueName: \"kubernetes.io/projected/b97719be-a0e9-42d0-a601-7a5e9d67976b-kube-api-access-zb94m\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:02:41.758846 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.758843    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-trusted-ca-bundle\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:02:41.758846 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.758852    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-service-ca\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:02:41.759118 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.758862    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b97719be-a0e9-42d0-a601-7a5e9d67976b-oauth-serving-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:02:41.759118 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.758870    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-oauth-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:02:41.759118 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:41.758880    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b97719be-a0e9-42d0-a601-7a5e9d67976b-console-serving-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:02:42.420699 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.420662    2566 generic.go:356] "Generic (PLEG): container finished" podID="b97719be-a0e9-42d0-a601-7a5e9d67976b" containerID="02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5" exitCode=0
Sep 09 20:02:42.420884 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.420714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799759b59f-vncgb" event={"ID":"b97719be-a0e9-42d0-a601-7a5e9d67976b","Type":"ContainerDied","Data":"02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5"}
Sep 09 20:02:42.420884 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.420722    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799759b59f-vncgb"
Sep 09 20:02:42.420884 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.420786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799759b59f-vncgb" event={"ID":"b97719be-a0e9-42d0-a601-7a5e9d67976b","Type":"ContainerDied","Data":"43d094bbaea3aee992e2c46e5aa4705b688cb5c798b08cf34d2c8ade6c5ed0fa"}
Sep 09 20:02:42.420884 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.420808    2566 scope.go:117] "RemoveContainer" containerID="02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5"
Sep 09 20:02:42.430844 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.430821    2566 scope.go:117] "RemoveContainer" containerID="02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5"
Sep 09 20:02:42.431213 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:02:42.431193    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5\": container with ID starting with 02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5 not found: ID does not exist" containerID="02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5"
Sep 09 20:02:42.431279 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.431223    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5"} err="failed to get container status \"02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5\": rpc error: code = NotFound desc = could not find container \"02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5\": container with ID starting with 02a991d6d503a0853bcfebf5993f24463690f8864079f36b3354bca7046b52b5 not found: ID does not exist"
Sep 09 20:02:42.444003 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.443977    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799759b59f-vncgb"]
Sep 09 20:02:42.449113 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.449088    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-799759b59f-vncgb"]
Sep 09 20:02:42.473052 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:42.473016    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97719be-a0e9-42d0-a601-7a5e9d67976b" path="/var/lib/kubelet/pods/b97719be-a0e9-42d0-a601-7a5e9d67976b/volumes"
Sep 09 20:02:45.331677 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:45.331648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djfmz"
Sep 09 20:02:50.363870 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:50.363831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-w5sg7"
Sep 09 20:02:50.364401 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:02:50.364191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-ckf8f"
Sep 09 20:03:16.742745 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:16.742704    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:16.802791 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:16.802752    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:39.602791 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:39.602751    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:45.233388 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.233319    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:45.308721 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.308684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:45.343181 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.343152    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-nmxv2"]
Sep 09 20:03:45.343525 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.343511    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97719be-a0e9-42d0-a601-7a5e9d67976b" containerName="console"
Sep 09 20:03:45.343572 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.343526    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97719be-a0e9-42d0-a601-7a5e9d67976b" containerName="console"
Sep 09 20:03:45.343607 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.343576    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97719be-a0e9-42d0-a601-7a5e9d67976b" containerName="console"
Sep 09 20:03:45.346578 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.346559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.349547 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.349518    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:03:45.349672 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.349585    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:03:45.349672 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.349597    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:03:45.349672 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.349522    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hxxjb\""
Sep 09 20:03:45.356039 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.356018    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nmxv2"]
Sep 09 20:03:45.417775 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.417737    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:45.517287 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.517194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5f94662-17ae-4671-9bc3-89c2334927e6-data\") pod \"seaweedfs-64568bcd49-nmxv2\" (UID: \"c5f94662-17ae-4671-9bc3-89c2334927e6\") " pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.517287 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.517252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvcdr\" (UniqueName: \"kubernetes.io/projected/c5f94662-17ae-4671-9bc3-89c2334927e6-kube-api-access-pvcdr\") pod \"seaweedfs-64568bcd49-nmxv2\" (UID: \"c5f94662-17ae-4671-9bc3-89c2334927e6\") " pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.555132 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.555092    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:45.617751 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.617697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvcdr\" (UniqueName: \"kubernetes.io/projected/c5f94662-17ae-4671-9bc3-89c2334927e6-kube-api-access-pvcdr\") pod \"seaweedfs-64568bcd49-nmxv2\" (UID: \"c5f94662-17ae-4671-9bc3-89c2334927e6\") " pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.617965 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.617821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5f94662-17ae-4671-9bc3-89c2334927e6-data\") pod \"seaweedfs-64568bcd49-nmxv2\" (UID: \"c5f94662-17ae-4671-9bc3-89c2334927e6\") " pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.618218 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.618197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5f94662-17ae-4671-9bc3-89c2334927e6-data\") pod \"seaweedfs-64568bcd49-nmxv2\" (UID: \"c5f94662-17ae-4671-9bc3-89c2334927e6\") " pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.630464 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.630430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvcdr\" (UniqueName: \"kubernetes.io/projected/c5f94662-17ae-4671-9bc3-89c2334927e6-kube-api-access-pvcdr\") pod \"seaweedfs-64568bcd49-nmxv2\" (UID: \"c5f94662-17ae-4671-9bc3-89c2334927e6\") " pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.655581 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.655544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:45.658315 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.658298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:45.931809 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:45.931709    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nmxv2"]
Sep 09 20:03:45.934492 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:03:45.934464    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f94662_17ae_4671_9bc3_89c2334927e6.slice/crio-cc177238f9be6ac87ed0a5f2942f2f909df17403929467189ac11eafcf236d2b WatchSource:0}: Error finding container cc177238f9be6ac87ed0a5f2942f2f909df17403929467189ac11eafcf236d2b: Status 404 returned error can't find the container with id cc177238f9be6ac87ed0a5f2942f2f909df17403929467189ac11eafcf236d2b
Sep 09 20:03:46.281389 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:46.281345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:46.723210 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:46.723176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nmxv2" event={"ID":"c5f94662-17ae-4671-9bc3-89c2334927e6","Type":"ContainerStarted","Data":"cc177238f9be6ac87ed0a5f2942f2f909df17403929467189ac11eafcf236d2b"}
Sep 09 20:03:47.633149 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:47.633115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:47.736246 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:47.736192    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:49.742918 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:49.742872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nmxv2" event={"ID":"c5f94662-17ae-4671-9bc3-89c2334927e6","Type":"ContainerStarted","Data":"6893e95107c114bdac29bba1f6503341d839d4f7cdc338af77e6e18ecaeb999b"}
Sep 09 20:03:49.743344 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:49.743170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:03:49.761557 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:49.761500    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-nmxv2" podStartSLOduration=1.473392507 podStartE2EDuration="4.761482299s" totalImagesPullingTime="3.288089792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:49.760382211 +0000 UTC m=+517.787913195" watchObservedRunningTime="2026-09-09 20:03:49.761482299 +0000 UTC m=+517.789013285"
Sep 09 20:03:55.749705 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:03:55.749675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-nmxv2"
Sep 09 20:04:22.444164 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:04:22.444077    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:22.542555 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:04:22.542516    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:55.920150 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:04:55.920090    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:02.856786 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.856752    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-9hf86"]
Sep 09 20:05:02.860335 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.860312    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:02.863308 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.863285    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:05:02.863560 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.863538    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-fj9xx\""
Sep 09 20:05:02.875338 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.875304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9hf86"]
Sep 09 20:05:02.877029 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.877005    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-gxfqb"]
Sep 09 20:05:02.880443 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.880421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:02.883401 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.883376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6m48c\""
Sep 09 20:05:02.883539 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.883383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:05:02.883672 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.883657    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:05:02.891613 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.891228    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-gxfqb"]
Sep 09 20:05:02.898996 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.898967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-kube-api-access-xdfwj\") pod \"model-serving-api-7b954ffc95-9hf86\" (UID: \"6d357f1d-90f1-4f3f-99e8-486cf754a8ec\") " pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:02.899164 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.899075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-tls-certs\") pod \"model-serving-api-7b954ffc95-9hf86\" (UID: \"6d357f1d-90f1-4f3f-99e8-486cf754a8ec\") " pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:02.999990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.999933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-metrics-certs\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:02.999990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:02.999995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clxml\" (UniqueName: \"kubernetes.io/projected/a5a8996a-eec1-43e4-97e4-a2b337f6012c-kube-api-access-clxml\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.000255 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.000049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-kube-api-access-xdfwj\") pod \"model-serving-api-7b954ffc95-9hf86\" (UID: \"6d357f1d-90f1-4f3f-99e8-486cf754a8ec\") " pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:03.000255 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.000129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-cert\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.000255 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.000222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-tls-certs\") pod \"model-serving-api-7b954ffc95-9hf86\" (UID: \"6d357f1d-90f1-4f3f-99e8-486cf754a8ec\") " pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:03.000449 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:05:03.000349    2566 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 20:05:03.000508 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:05:03.000454    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-tls-certs podName:6d357f1d-90f1-4f3f-99e8-486cf754a8ec nodeName:}" failed. No retries permitted until 2026-09-09 20:05:03.500431675 +0000 UTC m=+591.527962657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-tls-certs") pod "model-serving-api-7b954ffc95-9hf86" (UID: "6d357f1d-90f1-4f3f-99e8-486cf754a8ec") : secret "model-serving-api-tls" not found
Sep 09 20:05:03.011948 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.011918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-kube-api-access-xdfwj\") pod \"model-serving-api-7b954ffc95-9hf86\" (UID: \"6d357f1d-90f1-4f3f-99e8-486cf754a8ec\") " pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:03.100888 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.100847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-metrics-certs\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.101072 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.100898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clxml\" (UniqueName: \"kubernetes.io/projected/a5a8996a-eec1-43e4-97e4-a2b337f6012c-kube-api-access-clxml\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.101072 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.100938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-cert\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.101072 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:05:03.101014    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 09 20:05:03.101252 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:05:03.101096    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 20:05:03.101252 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:05:03.101099    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-metrics-certs podName:a5a8996a-eec1-43e4-97e4-a2b337f6012c nodeName:}" failed. No retries permitted until 2026-09-09 20:05:03.601080792 +0000 UTC m=+591.628611771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-metrics-certs") pod "odh-model-controller-7b966d7984-gxfqb" (UID: "a5a8996a-eec1-43e4-97e4-a2b337f6012c") : secret "odh-model-controller-metrics-tls" not found
Sep 09 20:05:03.101252 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:05:03.101155    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-cert podName:a5a8996a-eec1-43e4-97e4-a2b337f6012c nodeName:}" failed. No retries permitted until 2026-09-09 20:05:03.601140196 +0000 UTC m=+591.628671165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-cert") pod "odh-model-controller-7b966d7984-gxfqb" (UID: "a5a8996a-eec1-43e4-97e4-a2b337f6012c") : secret "odh-model-controller-webhook-cert" not found
Sep 09 20:05:03.113082 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.113011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clxml\" (UniqueName: \"kubernetes.io/projected/a5a8996a-eec1-43e4-97e4-a2b337f6012c-kube-api-access-clxml\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.117219 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.117189    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:03.212982 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.212927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:03.312934 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.312898    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:03.505027 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.504992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-tls-certs\") pod \"model-serving-api-7b954ffc95-9hf86\" (UID: \"6d357f1d-90f1-4f3f-99e8-486cf754a8ec\") " pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:03.507567 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.507535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d357f1d-90f1-4f3f-99e8-486cf754a8ec-tls-certs\") pod \"model-serving-api-7b954ffc95-9hf86\" (UID: \"6d357f1d-90f1-4f3f-99e8-486cf754a8ec\") " pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:03.606236 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.606150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-metrics-certs\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.606236 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.606230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-cert\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.607179 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.607152    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:03.608864 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.608841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-cert\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.608965 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.608878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a8996a-eec1-43e4-97e4-a2b337f6012c-metrics-certs\") pod \"odh-model-controller-7b966d7984-gxfqb\" (UID: \"a5a8996a-eec1-43e4-97e4-a2b337f6012c\") " pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:03.710281 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.710242    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:03.773223 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.773132    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:03.795509 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:03.795477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:04.057732 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:04.057709    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9hf86"]
Sep 09 20:05:04.059900 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:05:04.059866    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d357f1d_90f1_4f3f_99e8_486cf754a8ec.slice/crio-c679e442620f1df538e50172fb467ac0e81fa4b76857ae4628e4fd732496ad30 WatchSource:0}: Error finding container c679e442620f1df538e50172fb467ac0e81fa4b76857ae4628e4fd732496ad30: Status 404 returned error can't find the container with id c679e442620f1df538e50172fb467ac0e81fa4b76857ae4628e4fd732496ad30
Sep 09 20:05:04.069129 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:04.069104    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:04.083228 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:04.083202    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-gxfqb"]
Sep 09 20:05:04.089731 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:04.089700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9hf86" event={"ID":"6d357f1d-90f1-4f3f-99e8-486cf754a8ec","Type":"ContainerStarted","Data":"c679e442620f1df538e50172fb467ac0e81fa4b76857ae4628e4fd732496ad30"}
Sep 09 20:05:04.131449 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:04.131409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:05.097622 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:05.097572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-gxfqb" event={"ID":"a5a8996a-eec1-43e4-97e4-a2b337f6012c","Type":"ContainerStarted","Data":"2f1ba7ad5da60881184609cba43382ca5fa424ad1d8fa15073e6c415cdd0850d"}
Sep 09 20:05:07.112918 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:07.112814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-gxfqb" event={"ID":"a5a8996a-eec1-43e4-97e4-a2b337f6012c","Type":"ContainerStarted","Data":"4d2923cc96e8248d21eac2dbb7fa2bb25f8ef474405442a7c02fe6ff26bb9275"}
Sep 09 20:05:07.113354 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:07.113097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:07.132056 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:07.131992    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-gxfqb" podStartSLOduration=2.425466376 podStartE2EDuration="5.131971096s" totalImagesPullingTime="2.70650472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:05:07.131136785 +0000 UTC m=+595.158667771" watchObservedRunningTime="2026-09-09 20:05:07.131971096 +0000 UTC m=+595.159502081"
Sep 09 20:05:08.118495 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:08.118461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9hf86" event={"ID":"6d357f1d-90f1-4f3f-99e8-486cf754a8ec","Type":"ContainerStarted","Data":"5464589c5372dd6d9c72ae4a1a4ed57318a7988f50777cfb976c8a1bfccc9655"}
Sep 09 20:05:08.139473 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:08.139415    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-9hf86" podStartSLOduration=2.514759528 podStartE2EDuration="6.139395663s" totalImagesPullingTime="3.624636135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:05:08.136611374 +0000 UTC m=+596.164142396" watchObservedRunningTime="2026-09-09 20:05:08.139395663 +0000 UTC m=+596.166926675"
Sep 09 20:05:09.123900 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:09.123871    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:12.438539 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:12.438512    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:05:12.439183 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:12.439164    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:05:18.074423 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:18.074379    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:18.121272 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:18.121239    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-gxfqb"
Sep 09 20:05:20.132600 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:20.132567    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-9hf86"
Sep 09 20:05:58.027740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:05:58.027696    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:09.304214 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:09.304160    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:19.852240 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:19.852199    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:37.521373 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:37.521330    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:37.621990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:37.621959    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:51.771787 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.771745    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:51.837978 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.837946    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"]
Sep 09 20:06:51.841790 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.841771    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:51.844865 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.844841    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:06:51.845012 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.844973    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:06:51.845086 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.844908    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:06:51.845086 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.844908    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:06:51.845086 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.844909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2b317-predictor-serving-cert\""
Sep 09 20:06:51.845250 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.844909    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2b317-kube-rbac-proxy-sar-config\""
Sep 09 20:06:51.846021 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.846005    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:06:51.854832 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.854803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"]
Sep 09 20:06:51.958047 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.958010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:51.958228 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.958073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:51.958228 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.958099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:51.958228 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.958129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:51.958228 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.958169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmf4\" (UniqueName: \"kubernetes.io/projected/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-api-access-cqmf4\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:51.965141 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:51.965089    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:52.040161 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.040060    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:52.058951 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.058908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.059161 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.058964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.059240 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.059216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.059304 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.059284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.059378 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.059338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmf4\" (UniqueName: \"kubernetes.io/projected/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-api-access-cqmf4\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.059652 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.059625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.059854 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.059780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.059983 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.059943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.061852 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.061826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.072780 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.072750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmf4\" (UniqueName: \"kubernetes.io/projected/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-api-access-cqmf4\") pod \"isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.115687 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.115630    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:52.153239 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.153200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:06:52.339088 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.339000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:52.442337 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.442310    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"]
Sep 09 20:06:52.444573 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:06:52.444546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56977ae4_71a0_4981_a8dd_c4eec5ea4b86.slice/crio-5e46f1c70f2d02af8aeb0fde40b9317c07d5ec2d13e7b32abe36d41d49411606 WatchSource:0}: Error finding container 5e46f1c70f2d02af8aeb0fde40b9317c07d5ec2d13e7b32abe36d41d49411606: Status 404 returned error can't find the container with id 5e46f1c70f2d02af8aeb0fde40b9317c07d5ec2d13e7b32abe36d41d49411606
Sep 09 20:06:52.607739 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.607646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerStarted","Data":"5e46f1c70f2d02af8aeb0fde40b9317c07d5ec2d13e7b32abe36d41d49411606"}
Sep 09 20:06:52.755738 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:52.755701    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:57.637250 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:06:57.637214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerStarted","Data":"647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0"}
Sep 09 20:07:00.651760 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:00.651727    2566 generic.go:356] "Generic (PLEG): container finished" podID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerID="647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0" exitCode=0
Sep 09 20:07:00.652128 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:00.651790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerDied","Data":"647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0"}
Sep 09 20:07:14.742167 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:14.742120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerStarted","Data":"9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e"}
Sep 09 20:07:16.755631 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:16.755590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerStarted","Data":"4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3"}
Sep 09 20:07:19.773990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:19.773950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerStarted","Data":"6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d"}
Sep 09 20:07:19.774498 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:19.774071    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:07:19.774498 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:19.774126    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:07:19.775651 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:19.775619    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:07:19.804165 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:19.804111    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podStartSLOduration=2.254482111 podStartE2EDuration="28.80409129s" totalImagesPullingTime="23.00821271s" totalInitContainerRuntime="3.541396469s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:51 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:19.800053654 +0000 UTC m=+727.827584652" watchObservedRunningTime="2026-09-09 20:07:19.80409129 +0000 UTC m=+727.831622277"
Sep 09 20:07:20.778882 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:20.778837    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:07:20.779334 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:20.778988    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:07:20.780285 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:20.780251    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:07:21.784707 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:21.784659    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:07:21.785203 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:21.785146    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:07:21.788666 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:21.788643    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:07:22.789030 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:22.788990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:07:22.789472 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:22.789354    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:07:32.788905 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:32.788852    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:07:32.789520 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:32.789494    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:07:42.789239 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:42.789190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:07:42.789793 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:42.789767    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:07:52.789086 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:52.789034    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:07:52.789621 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:07:52.789579    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:02.788942 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:02.788889    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:08:02.789476 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:02.789350    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:12.789785 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:12.789732    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:08:12.790342 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:12.790295    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:22.789796 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:22.789741    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:08:22.790326 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:22.790283    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:23.464250 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:23.464205    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:08:23.464550 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:23.464524    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:33.465038 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:33.464946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:08:33.465463 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:33.465442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:08:41.781047 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.781011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:41.842911 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.842877    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"]
Sep 09 20:08:41.846650 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.846630    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.848606 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.848575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxhph\" (UniqueName: \"kubernetes.io/projected/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-api-access-vxhph\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.848740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.848609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.848740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.848632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.848740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.848687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.848740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.848731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.849715 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.849692    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:08:41.849801 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.849700    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:08:41.857855 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.857829    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"]
Sep 09 20:08:41.949966 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.949910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.950170 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.950042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxhph\" (UniqueName: \"kubernetes.io/projected/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-api-access-vxhph\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.950170 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.950071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.950269 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.950217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.950306 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.950281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.950552 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.950530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.950781 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.950763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.950836 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.950805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.952918 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.952896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:41.959483 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:41.959441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxhph\" (UniqueName: \"kubernetes.io/projected/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-api-access-vxhph\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5rr9b\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:42.158385 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:42.158268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:42.451869 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:42.451773    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"]
Sep 09 20:08:42.454590 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:08:42.454556    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dbc8bb_88dd_42c0_bf81_6dc2026194e5.slice/crio-2c68c4bb407e73c62d0abf31c0b7ef3ae5365edfdcdb531a0b736c1178a87f34 WatchSource:0}: Error finding container 2c68c4bb407e73c62d0abf31c0b7ef3ae5365edfdcdb531a0b736c1178a87f34: Status 404 returned error can't find the container with id 2c68c4bb407e73c62d0abf31c0b7ef3ae5365edfdcdb531a0b736c1178a87f34
Sep 09 20:08:43.168496 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:43.168460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerStarted","Data":"971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944"}
Sep 09 20:08:43.168880 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:43.168519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerStarted","Data":"2c68c4bb407e73c62d0abf31c0b7ef3ae5365edfdcdb531a0b736c1178a87f34"}
Sep 09 20:08:46.814402 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:46.814353    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"]
Sep 09 20:08:46.814844 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:46.814734    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" containerID="cri-o://9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e" gracePeriod=30
Sep 09 20:08:46.814844 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:46.814754    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy" containerID="cri-o://4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3" gracePeriod=30
Sep 09 20:08:46.814844 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:46.814808    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" containerID="cri-o://6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d" gracePeriod=30
Sep 09 20:08:46.849244 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:46.849213    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:46.950890 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:46.950852    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:47.066728 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:47.066640    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:47.193617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:47.193585    2566 generic.go:356] "Generic (PLEG): container finished" podID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerID="4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3" exitCode=2
Sep 09 20:08:47.193813 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:47.193667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerDied","Data":"4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3"}
Sep 09 20:08:47.195413 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:47.195386    2566 generic.go:356] "Generic (PLEG): container finished" podID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerID="971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944" exitCode=0
Sep 09 20:08:47.195528 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:47.195459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerDied","Data":"971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944"}
Sep 09 20:08:47.339698 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:47.339664    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:47.728655 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:47.728629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:48.202896 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:48.202810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerStarted","Data":"fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2"}
Sep 09 20:08:48.202896 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:48.202874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerStarted","Data":"9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db"}
Sep 09 20:08:48.203316 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:48.203146    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:48.203401 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:48.203311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:48.204948 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:48.204921    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:08:48.226108 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:48.226056    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podStartSLOduration=3.3916427000000002 podStartE2EDuration="7.226039001s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.834396301s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:08:48.225038143 +0000 UTC m=+816.252569142" watchObservedRunningTime="2026-09-09 20:08:48.226039001 +0000 UTC m=+816.253569987"
Sep 09 20:08:49.209483 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:49.209445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:08:51.785406 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:51.785327    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 09 20:08:52.225556 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:52.225515    2566 generic.go:356] "Generic (PLEG): container finished" podID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerID="9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e" exitCode=0
Sep 09 20:08:52.225556 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:52.225556    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerDied","Data":"9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e"}
Sep 09 20:08:53.464690 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:53.464641    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:53.465842 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:53.465812    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:08:54.214195 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:54.214165    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:08:54.214769 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:54.214740    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:08:56.784934 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:08:56.784880    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 09 20:09:01.785788 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:01.785742    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 09 20:09:01.786246 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:01.785871    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:09:03.464824 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:03.464783    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:03.465974 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:03.465940    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:09:04.215182 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:04.215142    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:09:06.785711 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:06.785660    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 09 20:09:11.785355 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:11.785305    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 09 20:09:13.464631 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:13.464584    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:13.465032 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:13.464700    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:09:13.465812 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:13.465788    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:09:13.465894 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:13.465881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:09:14.215118 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:14.215080    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:09:16.785235 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:16.785172    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 09 20:09:16.978408 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:16.978379    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:09:17.074602 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.074504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-cabundle-cert\") pod \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") "
Sep 09 20:09:17.074602 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.074547    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqmf4\" (UniqueName: \"kubernetes.io/projected/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-api-access-cqmf4\") pod \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") "
Sep 09 20:09:17.074602 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.074592    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-rbac-proxy-sar-config\") pod \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") "
Sep 09 20:09:17.074902 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.074658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-proxy-tls\") pod \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") "
Sep 09 20:09:17.074902 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.074693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kserve-provision-location\") pod \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\" (UID: \"56977ae4-71a0-4981-a8dd-c4eec5ea4b86\") "
Sep 09 20:09:17.075009 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.074957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56977ae4-71a0-4981-a8dd-c4eec5ea4b86" (UID: "56977ae4-71a0-4981-a8dd-c4eec5ea4b86"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:09:17.075009 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.074989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56977ae4-71a0-4981-a8dd-c4eec5ea4b86" (UID: "56977ae4-71a0-4981-a8dd-c4eec5ea4b86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:09:17.075124 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.075035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56977ae4-71a0-4981-a8dd-c4eec5ea4b86" (UID: "56977ae4-71a0-4981-a8dd-c4eec5ea4b86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:09:17.077096 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.077070    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56977ae4-71a0-4981-a8dd-c4eec5ea4b86" (UID: "56977ae4-71a0-4981-a8dd-c4eec5ea4b86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:09:17.077186 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.077071    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-api-access-cqmf4" (OuterVolumeSpecName: "kube-api-access-cqmf4") pod "56977ae4-71a0-4981-a8dd-c4eec5ea4b86" (UID: "56977ae4-71a0-4981-a8dd-c4eec5ea4b86"). InnerVolumeSpecName "kube-api-access-cqmf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:09:17.175856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.175808    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:09:17.175856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.175848    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmf4\" (UniqueName: \"kubernetes.io/projected/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-api-access-cqmf4\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:09:17.175856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.175860    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:09:17.175856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.175870    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:09:17.176221 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.175879    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56977ae4-71a0-4981-a8dd-c4eec5ea4b86-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:09:17.348024 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.347937    2566 generic.go:356] "Generic (PLEG): container finished" podID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerID="6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d" exitCode=0
Sep 09 20:09:17.348024 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.348006    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"
Sep 09 20:09:17.348211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.348016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerDied","Data":"6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d"}
Sep 09 20:09:17.348211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.348097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4" event={"ID":"56977ae4-71a0-4981-a8dd-c4eec5ea4b86","Type":"ContainerDied","Data":"5e46f1c70f2d02af8aeb0fde40b9317c07d5ec2d13e7b32abe36d41d49411606"}
Sep 09 20:09:17.348211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.348144    2566 scope.go:117] "RemoveContainer" containerID="6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d"
Sep 09 20:09:17.357505 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.357484    2566 scope.go:117] "RemoveContainer" containerID="4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3"
Sep 09 20:09:17.366107 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.366078    2566 scope.go:117] "RemoveContainer" containerID="9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e"
Sep 09 20:09:17.373006 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.372977    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"]
Sep 09 20:09:17.374796 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.374764    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2b317-predictor-6d7fb8646c-98hg4"]
Sep 09 20:09:17.375832 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.375816    2566 scope.go:117] "RemoveContainer" containerID="647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0"
Sep 09 20:09:17.384636 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.384618    2566 scope.go:117] "RemoveContainer" containerID="6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d"
Sep 09 20:09:17.384984 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:09:17.384955    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d\": container with ID starting with 6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d not found: ID does not exist" containerID="6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d"
Sep 09 20:09:17.385069 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.384999    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d"} err="failed to get container status \"6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d\": rpc error: code = NotFound desc = could not find container \"6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d\": container with ID starting with 6d03ae3d5d710c9257d76bfdf3f1daf25057535dd126cbf2b445e99acbad1d0d not found: ID does not exist"
Sep 09 20:09:17.385069 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.385025    2566 scope.go:117] "RemoveContainer" containerID="4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3"
Sep 09 20:09:17.385344 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:09:17.385324    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3\": container with ID starting with 4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3 not found: ID does not exist" containerID="4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3"
Sep 09 20:09:17.385434 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.385350    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3"} err="failed to get container status \"4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3\": rpc error: code = NotFound desc = could not find container \"4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3\": container with ID starting with 4a2fc0a44d7912b995805613e1c73febc1cf49720b7c711e5166513345604af3 not found: ID does not exist"
Sep 09 20:09:17.385434 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.385394    2566 scope.go:117] "RemoveContainer" containerID="9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e"
Sep 09 20:09:17.385657 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:09:17.385631    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e\": container with ID starting with 9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e not found: ID does not exist" containerID="9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e"
Sep 09 20:09:17.385749 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.385666    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e"} err="failed to get container status \"9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e\": rpc error: code = NotFound desc = could not find container \"9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e\": container with ID starting with 9072ddad82c8e1b449abcad5c4f1327055a8f95a3ad872ec20808091f9733f2e not found: ID does not exist"
Sep 09 20:09:17.385749 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.385690    2566 scope.go:117] "RemoveContainer" containerID="647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0"
Sep 09 20:09:17.385940 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:09:17.385924    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0\": container with ID starting with 647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0 not found: ID does not exist" containerID="647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0"
Sep 09 20:09:17.385994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:17.385948    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0"} err="failed to get container status \"647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0\": rpc error: code = NotFound desc = could not find container \"647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0\": container with ID starting with 647ffc48be8fa94db7a34e8d7a4dabfe93ab443afc125456d4aed07be23cbdb0 not found: ID does not exist"
Sep 09 20:09:18.471382 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:18.471330    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" path="/var/lib/kubelet/pods/56977ae4-71a0-4981-a8dd-c4eec5ea4b86/volumes"
Sep 09 20:09:24.215274 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:24.215232    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:09:34.215236 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:34.215192    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:09:44.214729 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:44.214689    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:09:54.215683 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:09:54.215638    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:10:04.216015 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:04.215931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:10:11.862171 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:11.862133    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"]
Sep 09 20:10:11.862648 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:11.862453    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" containerID="cri-o://9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db" gracePeriod=30
Sep 09 20:10:11.862648 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:11.862497    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kube-rbac-proxy" containerID="cri-o://fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2" gracePeriod=30
Sep 09 20:10:11.891159 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:11.891124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:12.470939 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:12.470905    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:10:12.473644 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:12.473617    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:10:12.610843 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:12.610810    2566 generic.go:356] "Generic (PLEG): container finished" podID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerID="fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2" exitCode=2
Sep 09 20:10:12.611024 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:12.610893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerDied","Data":"fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2"}
Sep 09 20:10:14.210692 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:14.210643    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 09 20:10:14.215033 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:14.215003    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:10:16.918333 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:16.918296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:10:17.000255 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.000223    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kserve-provision-location\") pod \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") "
Sep 09 20:10:17.000436 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.000269    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxhph\" (UniqueName: \"kubernetes.io/projected/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-api-access-vxhph\") pod \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") "
Sep 09 20:10:17.000436 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.000297    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-cabundle-cert\") pod \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") "
Sep 09 20:10:17.000700 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.000605    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65dbc8bb-88dd-42c0-bf81-6dc2026194e5" (UID: "65dbc8bb-88dd-42c0-bf81-6dc2026194e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:10:17.000775 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.000750    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65dbc8bb-88dd-42c0-bf81-6dc2026194e5" (UID: "65dbc8bb-88dd-42c0-bf81-6dc2026194e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:17.002865 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.002837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-api-access-vxhph" (OuterVolumeSpecName: "kube-api-access-vxhph") pod "65dbc8bb-88dd-42c0-bf81-6dc2026194e5" (UID: "65dbc8bb-88dd-42c0-bf81-6dc2026194e5"). InnerVolumeSpecName "kube-api-access-vxhph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:10:17.039492 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.039455    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:17.096819 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.096778    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"]
Sep 09 20:10:17.097327 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097310    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="storage-initializer"
Sep 09 20:10:17.097411 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097331    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="storage-initializer"
Sep 09 20:10:17.097411 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097347    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy"
Sep 09 20:10:17.097411 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097372    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy"
Sep 09 20:10:17.097411 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097382    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container"
Sep 09 20:10:17.097411 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097391    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097416    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="storage-initializer"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097425    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="storage-initializer"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097436    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kube-rbac-proxy"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097444    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kube-rbac-proxy"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097458    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097467    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097478    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097486    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097557    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="agent"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097568    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kube-rbac-proxy"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097579    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerName="kserve-container"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097589    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kserve-container"
Sep 09 20:10:17.097594 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.097598    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="56977ae4-71a0-4981-a8dd-c4eec5ea4b86" containerName="kube-rbac-proxy"
Sep 09 20:10:17.101071 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.101042    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-proxy-tls\") pod \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") "
Sep 09 20:10:17.101196 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.101145    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-rbac-proxy-sar-config\") pod \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\" (UID: \"65dbc8bb-88dd-42c0-bf81-6dc2026194e5\") "
Sep 09 20:10:17.101286 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.101266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.101699 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.101444    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:10:17.101699 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.101470    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxhph\" (UniqueName: \"kubernetes.io/projected/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-api-access-vxhph\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:10:17.101699 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.101484    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:10:17.101699 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.101524    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65dbc8bb-88dd-42c0-bf81-6dc2026194e5" (UID: "65dbc8bb-88dd-42c0-bf81-6dc2026194e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:17.103541 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.103518    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65dbc8bb-88dd-42c0-bf81-6dc2026194e5" (UID: "65dbc8bb-88dd-42c0-bf81-6dc2026194e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:10:17.104031 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.104005    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 09 20:10:17.104147 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.104111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:10:17.113995 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.113971    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"]
Sep 09 20:10:17.121153 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.121124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:17.168198 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.168165    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"]
Sep 09 20:10:17.172140 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.172116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.175068 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.175045    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:10:17.182136 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.182111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"]
Sep 09 20:10:17.201820 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.201792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa2cf86-1a49-44c9-9029-d707e9224810-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.201820 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.201834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.201820 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.201856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.201881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa2cf86-1a49-44c9-9029-d707e9224810-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.201921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.201946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.201991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.202010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.202027    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.202034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zgg\" (UniqueName: \"kubernetes.io/projected/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-api-access-b5zgg\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.202096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5x79\" (UniqueName: \"kubernetes.io/projected/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-api-access-b5x79\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.202155    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:10:17.202211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.202170    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65dbc8bb-88dd-42c0-bf81-6dc2026194e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:10:17.303296 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa2cf86-1a49-44c9-9029-d707e9224810-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.303296 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.303296 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.303296 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa2cf86-1a49-44c9-9029-d707e9224810-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zgg\" (UniqueName: \"kubernetes.io/projected/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-api-access-b5zgg\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x79\" (UniqueName: \"kubernetes.io/projected/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-api-access-b5x79\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:10:17.303412    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 09 20:10:17.303659 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:10:17.303611    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls podName:ef1a4376-35d8-478b-be46-ad56ffd85ffa nodeName:}" failed. No retries permitted until 2026-09-09 20:10:17.803587079 +0000 UTC m=+905.831118051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls") pod "isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" (UID: "ef1a4376-35d8-478b-be46-ad56ffd85ffa") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 09 20:10:17.304162 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.303820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa2cf86-1a49-44c9-9029-d707e9224810-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.304162 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.304047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.304282 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.304260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.304324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.304301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.304423 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.304407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.304687 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.304671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.306422 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.306405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa2cf86-1a49-44c9-9029-d707e9224810-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.313461 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.313430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x79\" (UniqueName: \"kubernetes.io/projected/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-api-access-b5x79\") pod \"isvc-canary-stop-predictor-5d5b899946-bdq49\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.313568 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.313489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zgg\" (UniqueName: \"kubernetes.io/projected/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-api-access-b5zgg\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.453168 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.453126    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:17.485309 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.485268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:17.636960 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.636867    2566 generic.go:356] "Generic (PLEG): container finished" podID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" containerID="9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db" exitCode=0
Sep 09 20:10:17.636960 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.636922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerDied","Data":"9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db"}
Sep 09 20:10:17.636960 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.636931    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"
Sep 09 20:10:17.637192 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.636983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b" event={"ID":"65dbc8bb-88dd-42c0-bf81-6dc2026194e5","Type":"ContainerDied","Data":"2c68c4bb407e73c62d0abf31c0b7ef3ae5365edfdcdb531a0b736c1178a87f34"}
Sep 09 20:10:17.637192 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.637012    2566 scope.go:117] "RemoveContainer" containerID="fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2"
Sep 09 20:10:17.645831 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.645807    2566 scope.go:117] "RemoveContainer" containerID="9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db"
Sep 09 20:10:17.656837 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.656814    2566 scope.go:117] "RemoveContainer" containerID="971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944"
Sep 09 20:10:17.662192 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.662161    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"]
Sep 09 20:10:17.665235 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.665207    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5rr9b"]
Sep 09 20:10:17.667669 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.667653    2566 scope.go:117] "RemoveContainer" containerID="fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2"
Sep 09 20:10:17.668004 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:10:17.667986    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2\": container with ID starting with fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2 not found: ID does not exist" containerID="fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2"
Sep 09 20:10:17.668045 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.668015    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2"} err="failed to get container status \"fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2\": rpc error: code = NotFound desc = could not find container \"fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2\": container with ID starting with fa558ef48c61ff994fad17383e85f199ab8c5c3b085123565201934d1fd62be2 not found: ID does not exist"
Sep 09 20:10:17.668045 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.668035    2566 scope.go:117] "RemoveContainer" containerID="9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db"
Sep 09 20:10:17.668311 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:10:17.668292    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db\": container with ID starting with 9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db not found: ID does not exist" containerID="9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db"
Sep 09 20:10:17.668406 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.668320    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db"} err="failed to get container status \"9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db\": rpc error: code = NotFound desc = could not find container \"9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db\": container with ID starting with 9861326fb7d6acf1663fb537369c25b9729f3cc84509e764d34184b2028e09db not found: ID does not exist"
Sep 09 20:10:17.668406 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.668343    2566 scope.go:117] "RemoveContainer" containerID="971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944"
Sep 09 20:10:17.668684 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:10:17.668660    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944\": container with ID starting with 971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944 not found: ID does not exist" containerID="971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944"
Sep 09 20:10:17.668728 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.668691    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944"} err="failed to get container status \"971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944\": rpc error: code = NotFound desc = could not find container \"971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944\": container with ID starting with 971ee540aa194c2d8dc14415bc3960e47b9aebdde87141e5282226de5fe45944 not found: ID does not exist"
Sep 09 20:10:17.762959 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.762932    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"]
Sep 09 20:10:17.765657 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:10:17.765628    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa2cf86_1a49_44c9_9029_d707e9224810.slice/crio-13038b82f34925ddbfd1dbf2ccdccc01ddcf2185f81b1f2acfcd05a11435ff71 WatchSource:0}: Error finding container 13038b82f34925ddbfd1dbf2ccdccc01ddcf2185f81b1f2acfcd05a11435ff71: Status 404 returned error can't find the container with id 13038b82f34925ddbfd1dbf2ccdccc01ddcf2185f81b1f2acfcd05a11435ff71
Sep 09 20:10:17.809249 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.809214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:17.811964 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:17.811942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:18.020566 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:18.020520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:18.309118 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:18.308848    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"]
Sep 09 20:10:18.311755 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:10:18.311723    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1a4376_35d8_478b_be46_ad56ffd85ffa.slice/crio-4af3012fa996229512abfde524abfce5a55060b50d51a6b211d21b4554237c7d WatchSource:0}: Error finding container 4af3012fa996229512abfde524abfce5a55060b50d51a6b211d21b4554237c7d: Status 404 returned error can't find the container with id 4af3012fa996229512abfde524abfce5a55060b50d51a6b211d21b4554237c7d
Sep 09 20:10:18.470691 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:18.470658    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65dbc8bb-88dd-42c0-bf81-6dc2026194e5" path="/var/lib/kubelet/pods/65dbc8bb-88dd-42c0-bf81-6dc2026194e5/volumes"
Sep 09 20:10:18.644378 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:18.644272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerStarted","Data":"455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db"}
Sep 09 20:10:18.644378 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:18.644345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerStarted","Data":"13038b82f34925ddbfd1dbf2ccdccc01ddcf2185f81b1f2acfcd05a11435ff71"}
Sep 09 20:10:18.647064 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:18.647027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerStarted","Data":"e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1"}
Sep 09 20:10:18.647216 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:18.647088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerStarted","Data":"4af3012fa996229512abfde524abfce5a55060b50d51a6b211d21b4554237c7d"}
Sep 09 20:10:21.662888 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:21.662852    2566 generic.go:356] "Generic (PLEG): container finished" podID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerID="455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db" exitCode=0
Sep 09 20:10:21.663262 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:21.662928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerDied","Data":"455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db"}
Sep 09 20:10:22.668835 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.668802    2566 generic.go:356] "Generic (PLEG): container finished" podID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerID="e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1" exitCode=0
Sep 09 20:10:22.669275 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.668877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerDied","Data":"e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1"}
Sep 09 20:10:22.671211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.671186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerStarted","Data":"f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548"}
Sep 09 20:10:22.671316 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.671238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerStarted","Data":"0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70"}
Sep 09 20:10:22.671580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.671560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:22.671671 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.671617    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:22.673012 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.672990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:10:22.712220 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:22.712154    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podStartSLOduration=1.934804856 podStartE2EDuration="5.712134091s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777329235s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:22.710980763 +0000 UTC m=+910.738511747" watchObservedRunningTime="2026-09-09 20:10:22.712134091 +0000 UTC m=+910.739665077"
Sep 09 20:10:23.678479 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:23.678439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerStarted","Data":"7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d"}
Sep 09 20:10:23.678885 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:23.678518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerStarted","Data":"5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296"}
Sep 09 20:10:23.678885 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:23.678739    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:10:23.679031 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:23.678917    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:23.679074 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:23.679053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:23.680427 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:23.680397    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:10:23.701081 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:23.701022    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podStartSLOduration=2.992170186 podStartE2EDuration="6.701003165s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.708832979s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:23.698557591 +0000 UTC m=+911.726088577" watchObservedRunningTime="2026-09-09 20:10:23.701003165 +0000 UTC m=+911.728534150"
Sep 09 20:10:24.683569 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:24.683522    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:10:27.001503 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:27.001473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:27.151035 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:27.150989    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:28.684220 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:28.684182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:10:28.684665 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:28.684603    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:10:29.690304 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:29.690270    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:10:29.690848 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:29.690819    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:10:37.075647 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:37.075611    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:37.199729 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:37.199669    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:37.301672 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:37.301639    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:37.400528 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:37.400428    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:37.462612 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:37.462575    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:37.524032 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:37.523992    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:37.741572 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:37.741533    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:38.684763 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:38.684712    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:10:39.691619 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:39.691579    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:10:48.685274 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:48.685216    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:10:49.690815 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:49.690766    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:10:58.685279 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:58.685229    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:10:59.691183 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:10:59.691145    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:11:08.684950 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:08.684899    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:11:09.691763 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:09.691723    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:11:18.685338 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:18.685292    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:11:19.691066 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:19.691026    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:11:28.685534 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:28.685473    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:11:29.691078 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:29.691033    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:11:36.464931 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:36.464820    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:11:37.465557 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:37.465523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:11:46.470751 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:46.470720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:11:47.116311 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.116278    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"]
Sep 09 20:11:47.116552 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.116508    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container" containerID="cri-o://5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296" gracePeriod=30
Sep 09 20:11:47.116609 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.116563    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kube-rbac-proxy" containerID="cri-o://7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d" gracePeriod=30
Sep 09 20:11:47.134601 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.134567    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:47.176266 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.176227    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"]
Sep 09 20:11:47.176508 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.176481    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" containerID="cri-o://0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70" gracePeriod=30
Sep 09 20:11:47.176568 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.176529    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kube-rbac-proxy" containerID="cri-o://f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548" gracePeriod=30
Sep 09 20:11:47.230693 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.230653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:47.464903 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:47.464803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:11:48.101330 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:48.101292    2566 generic.go:356] "Generic (PLEG): container finished" podID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerID="7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d" exitCode=2
Sep 09 20:11:48.101726 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:48.101382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerDied","Data":"7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d"}
Sep 09 20:11:48.103546 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:48.103515    2566 generic.go:356] "Generic (PLEG): container finished" podID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerID="f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548" exitCode=2
Sep 09 20:11:48.103657 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:48.103573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerDied","Data":"f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548"}
Sep 09 20:11:48.679918 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:48.679872    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 09 20:11:49.684517 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:49.684465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:11:52.445480 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.445411    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:11:52.448120 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.448103    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:11:52.535191 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-cabundle-cert\") pod \"8fa2cf86-1a49-44c9-9029-d707e9224810\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") "
Sep 09 20:11:52.535424 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535206    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa2cf86-1a49-44c9-9029-d707e9224810-proxy-tls\") pod \"8fa2cf86-1a49-44c9-9029-d707e9224810\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") "
Sep 09 20:11:52.535424 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535233    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-cabundle-cert\") pod \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") "
Sep 09 20:11:52.535424 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535264    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-rbac-proxy-sar-config\") pod \"8fa2cf86-1a49-44c9-9029-d707e9224810\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") "
Sep 09 20:11:52.535424 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535324    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls\") pod \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") "
Sep 09 20:11:52.535424 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535348    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa2cf86-1a49-44c9-9029-d707e9224810-kserve-provision-location\") pod \"8fa2cf86-1a49-44c9-9029-d707e9224810\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") "
Sep 09 20:11:52.535424 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535420    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5x79\" (UniqueName: \"kubernetes.io/projected/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-api-access-b5x79\") pod \"8fa2cf86-1a49-44c9-9029-d707e9224810\" (UID: \"8fa2cf86-1a49-44c9-9029-d707e9224810\") "
Sep 09 20:11:52.535747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535470    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5zgg\" (UniqueName: \"kubernetes.io/projected/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-api-access-b5zgg\") pod \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") "
Sep 09 20:11:52.535747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-rbac-proxy-sar-config\") pod \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") "
Sep 09 20:11:52.535747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kserve-provision-location\") pod \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\" (UID: \"ef1a4376-35d8-478b-be46-ad56ffd85ffa\") "
Sep 09 20:11:52.535747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef1a4376-35d8-478b-be46-ad56ffd85ffa" (UID: "ef1a4376-35d8-478b-be46-ad56ffd85ffa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:52.535747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535691    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fa2cf86-1a49-44c9-9029-d707e9224810" (UID: "8fa2cf86-1a49-44c9-9029-d707e9224810"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:52.536002 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2cf86-1a49-44c9-9029-d707e9224810-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fa2cf86-1a49-44c9-9029-d707e9224810" (UID: "8fa2cf86-1a49-44c9-9029-d707e9224810"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:11:52.536002 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.535691    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fa2cf86-1a49-44c9-9029-d707e9224810" (UID: "8fa2cf86-1a49-44c9-9029-d707e9224810"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:52.536110 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.536002    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef1a4376-35d8-478b-be46-ad56ffd85ffa" (UID: "ef1a4376-35d8-478b-be46-ad56ffd85ffa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:11:52.536165 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.536107    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef1a4376-35d8-478b-be46-ad56ffd85ffa" (UID: "ef1a4376-35d8-478b-be46-ad56ffd85ffa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:52.536165 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.536131    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.536165 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.536153    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.536295 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.536167    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.536295 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.536183    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.536295 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.536198    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa2cf86-1a49-44c9-9029-d707e9224810-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.537687 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.537664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa2cf86-1a49-44c9-9029-d707e9224810-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fa2cf86-1a49-44c9-9029-d707e9224810" (UID: "8fa2cf86-1a49-44c9-9029-d707e9224810"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:11:52.537926 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.537905    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-api-access-b5zgg" (OuterVolumeSpecName: "kube-api-access-b5zgg") pod "ef1a4376-35d8-478b-be46-ad56ffd85ffa" (UID: "ef1a4376-35d8-478b-be46-ad56ffd85ffa"). InnerVolumeSpecName "kube-api-access-b5zgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:11:52.538132 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.538113    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef1a4376-35d8-478b-be46-ad56ffd85ffa" (UID: "ef1a4376-35d8-478b-be46-ad56ffd85ffa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:11:52.538188 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.538173    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-api-access-b5x79" (OuterVolumeSpecName: "kube-api-access-b5x79") pod "8fa2cf86-1a49-44c9-9029-d707e9224810" (UID: "8fa2cf86-1a49-44c9-9029-d707e9224810"). InnerVolumeSpecName "kube-api-access-b5x79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:11:52.637044 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.637007    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa2cf86-1a49-44c9-9029-d707e9224810-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.637044 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.637034    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef1a4376-35d8-478b-be46-ad56ffd85ffa-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.637044 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.637045    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5x79\" (UniqueName: \"kubernetes.io/projected/8fa2cf86-1a49-44c9-9029-d707e9224810-kube-api-access-b5x79\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.637044 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.637054    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zgg\" (UniqueName: \"kubernetes.io/projected/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-api-access-b5zgg\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.637304 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:52.637063    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef1a4376-35d8-478b-be46-ad56ffd85ffa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:11:53.129686 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.129648    2566 generic.go:356] "Generic (PLEG): container finished" podID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerID="0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70" exitCode=0
Sep 09 20:11:53.129868 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.129714    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"
Sep 09 20:11:53.129868 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.129720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerDied","Data":"0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70"}
Sep 09 20:11:53.129868 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.129787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49" event={"ID":"8fa2cf86-1a49-44c9-9029-d707e9224810","Type":"ContainerDied","Data":"13038b82f34925ddbfd1dbf2ccdccc01ddcf2185f81b1f2acfcd05a11435ff71"}
Sep 09 20:11:53.129868 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.129816    2566 scope.go:117] "RemoveContainer" containerID="f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548"
Sep 09 20:11:53.132103 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.132077    2566 generic.go:356] "Generic (PLEG): container finished" podID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerID="5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296" exitCode=0
Sep 09 20:11:53.132213 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.132155    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"
Sep 09 20:11:53.132213 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.132166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerDied","Data":"5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296"}
Sep 09 20:11:53.132300 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.132234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6" event={"ID":"ef1a4376-35d8-478b-be46-ad56ffd85ffa","Type":"ContainerDied","Data":"4af3012fa996229512abfde524abfce5a55060b50d51a6b211d21b4554237c7d"}
Sep 09 20:11:53.140541 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.140517    2566 scope.go:117] "RemoveContainer" containerID="0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70"
Sep 09 20:11:53.149003 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.148983    2566 scope.go:117] "RemoveContainer" containerID="455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db"
Sep 09 20:11:53.154103 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.154081    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"]
Sep 09 20:11:53.158076 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.158058    2566 scope.go:117] "RemoveContainer" containerID="f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548"
Sep 09 20:11:53.158422 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:11:53.158401    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548\": container with ID starting with f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548 not found: ID does not exist" containerID="f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548"
Sep 09 20:11:53.158512 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.158435    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548"} err="failed to get container status \"f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548\": rpc error: code = NotFound desc = could not find container \"f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548\": container with ID starting with f459a36d9b68012e53865c4374c6a2ce093e51b8129de3d70e630c2f64021548 not found: ID does not exist"
Sep 09 20:11:53.158512 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.158460    2566 scope.go:117] "RemoveContainer" containerID="0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70"
Sep 09 20:11:53.158619 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.158599    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-bdq49"]
Sep 09 20:11:53.158743 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:11:53.158726    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70\": container with ID starting with 0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70 not found: ID does not exist" containerID="0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70"
Sep 09 20:11:53.158784 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.158751    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70"} err="failed to get container status \"0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70\": rpc error: code = NotFound desc = could not find container \"0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70\": container with ID starting with 0e5e0c011379a38169dc1fe61d919738de64a42da25dcc7989681aa8f59e6e70 not found: ID does not exist"
Sep 09 20:11:53.158784 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.158770    2566 scope.go:117] "RemoveContainer" containerID="455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db"
Sep 09 20:11:53.159095 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:11:53.159077    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db\": container with ID starting with 455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db not found: ID does not exist" containerID="455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db"
Sep 09 20:11:53.159153 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.159101    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db"} err="failed to get container status \"455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db\": rpc error: code = NotFound desc = could not find container \"455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db\": container with ID starting with 455bea5476ea48d764288b0de93ccfabb1a08e2136ac62c32f3b596a065030db not found: ID does not exist"
Sep 09 20:11:53.159153 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.159120    2566 scope.go:117] "RemoveContainer" containerID="7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d"
Sep 09 20:11:53.166679 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.166660    2566 scope.go:117] "RemoveContainer" containerID="5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296"
Sep 09 20:11:53.170528 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.170481    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"]
Sep 09 20:11:53.172174 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.172155    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-2g6t6"]
Sep 09 20:11:53.175217 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.175200    2566 scope.go:117] "RemoveContainer" containerID="e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1"
Sep 09 20:11:53.182871 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.182851    2566 scope.go:117] "RemoveContainer" containerID="7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d"
Sep 09 20:11:53.183150 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:11:53.183130    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d\": container with ID starting with 7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d not found: ID does not exist" containerID="7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d"
Sep 09 20:11:53.183205 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.183161    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d"} err="failed to get container status \"7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d\": rpc error: code = NotFound desc = could not find container \"7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d\": container with ID starting with 7eb5be2ee7f155790993a40056f51144a094bbdd3a58bbbaa0b55126a9eeed3d not found: ID does not exist"
Sep 09 20:11:53.183205 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.183180    2566 scope.go:117] "RemoveContainer" containerID="5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296"
Sep 09 20:11:53.183501 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:11:53.183482    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296\": container with ID starting with 5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296 not found: ID does not exist" containerID="5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296"
Sep 09 20:11:53.183579 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.183510    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296"} err="failed to get container status \"5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296\": rpc error: code = NotFound desc = could not find container \"5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296\": container with ID starting with 5100a9090f65030e843e0a698101c58cf55ab682a653db7223a2b765e43be296 not found: ID does not exist"
Sep 09 20:11:53.183579 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.183532    2566 scope.go:117] "RemoveContainer" containerID="e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1"
Sep 09 20:11:53.183813 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:11:53.183793    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1\": container with ID starting with e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1 not found: ID does not exist" containerID="e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1"
Sep 09 20:11:53.183853 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:53.183819    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1"} err="failed to get container status \"e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1\": rpc error: code = NotFound desc = could not find container \"e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1\": container with ID starting with e95c52761a78231e128e2dcd8a97162740c8dfa37121f0ffad834d940fb6cfd1 not found: ID does not exist"
Sep 09 20:11:54.470489 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:54.470459    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" path="/var/lib/kubelet/pods/8fa2cf86-1a49-44c9-9029-d707e9224810/volumes"
Sep 09 20:11:54.470977 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:54.470963    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" path="/var/lib/kubelet/pods/ef1a4376-35d8-478b-be46-ad56ffd85ffa/volumes"
Sep 09 20:11:57.323538 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.323496    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:57.352883 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.352851    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"]
Sep 09 20:11:57.353304 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353286    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container"
Sep 09 20:11:57.353304 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353305    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353323    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="storage-initializer"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353329    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="storage-initializer"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353338    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353343    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353349    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kube-rbac-proxy"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353354    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kube-rbac-proxy"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353380    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="storage-initializer"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353385    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="storage-initializer"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353395    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kube-rbac-proxy"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353399    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kube-rbac-proxy"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353449    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kserve-container"
Sep 09 20:11:57.353455 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353458    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kube-rbac-proxy"
Sep 09 20:11:57.353802 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353465    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa2cf86-1a49-44c9-9029-d707e9224810" containerName="kube-rbac-proxy"
Sep 09 20:11:57.353802 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.353471    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1a4376-35d8-478b-be46-ad56ffd85ffa" containerName="kserve-container"
Sep 09 20:11:57.357873 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.357853    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.360638 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.360611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:11:57.361039 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.361013    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-05a1b8-predictor-serving-cert\""
Sep 09 20:11:57.361158 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.361096    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-05a1b8-kube-rbac-proxy-sar-config\""
Sep 09 20:11:57.361158 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.361103    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:11:57.361158 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.361115    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:11:57.361158 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.361152    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:11:57.361420 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.361100    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:11:57.368135 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.368109    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"]
Sep 09 20:11:57.479908 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.479872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.480077 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.479911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ac1af84-12bb-48bf-84d3-6d241477127a-kserve-provision-location\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.480077 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.480002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ac1af84-12bb-48bf-84d3-6d241477127a-proxy-tls\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.480077 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.480040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgls6\" (UniqueName: \"kubernetes.io/projected/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-api-access-fgls6\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.480077 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.480061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-cabundle-cert\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.490526 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.490490    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:57.581146 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-cabundle-cert\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.581317 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.581317 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ac1af84-12bb-48bf-84d3-6d241477127a-kserve-provision-location\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.581317 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ac1af84-12bb-48bf-84d3-6d241477127a-proxy-tls\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.581317 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgls6\" (UniqueName: \"kubernetes.io/projected/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-api-access-fgls6\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.581673 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ac1af84-12bb-48bf-84d3-6d241477127a-kserve-provision-location\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.581968 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.582012 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.581946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-cabundle-cert\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.583816 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.583795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ac1af84-12bb-48bf-84d3-6d241477127a-proxy-tls\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.592868 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.592842    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgls6\" (UniqueName: \"kubernetes.io/projected/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-api-access-fgls6\") pod \"isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") " pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.670542 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.670500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:11:57.945954 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:57.945928    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"]
Sep 09 20:11:57.948238 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:11:57.948210    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac1af84_12bb_48bf_84d3_6d241477127a.slice/crio-71eae3eecbb2a47fb286308d03ea8c204e0feff8cb6f4c454a4bb43c431d732d WatchSource:0}: Error finding container 71eae3eecbb2a47fb286308d03ea8c204e0feff8cb6f4c454a4bb43c431d732d: Status 404 returned error can't find the container with id 71eae3eecbb2a47fb286308d03ea8c204e0feff8cb6f4c454a4bb43c431d732d
Sep 09 20:11:58.162330 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:58.162290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerStarted","Data":"31baf446cd7af3991e8b1baf0fbe86292d768fcfd297171df3364b172d707190"}
Sep 09 20:11:58.162519 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:11:58.162389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerStarted","Data":"71eae3eecbb2a47fb286308d03ea8c204e0feff8cb6f4c454a4bb43c431d732d"}
Sep 09 20:12:02.182697 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:02.182606    2566 generic.go:356] "Generic (PLEG): container finished" podID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerID="31baf446cd7af3991e8b1baf0fbe86292d768fcfd297171df3364b172d707190" exitCode=0
Sep 09 20:12:02.182697 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:02.182680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerDied","Data":"31baf446cd7af3991e8b1baf0fbe86292d768fcfd297171df3364b172d707190"}
Sep 09 20:12:03.189412 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:03.189355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerStarted","Data":"1835c38d2a3ee8c48328cf43b735c73f80cc5c85011e6d7d89bdd30ffbb9c8a7"}
Sep 09 20:12:03.189814 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:03.189444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerStarted","Data":"b0fecb3f97a787fcae1466b9b0984c7eb6d7659c3ad812560cc3d185588de783"}
Sep 09 20:12:03.189814 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:03.189701    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:12:03.213086 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:03.213031    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podStartSLOduration=2.405012791 podStartE2EDuration="6.213009791s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.807997s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:03.210916569 +0000 UTC m=+1011.238447557" watchObservedRunningTime="2026-09-09 20:12:03.213009791 +0000 UTC m=+1011.240540776"
Sep 09 20:12:04.194617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:04.194578    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:12:04.195830 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:04.195800    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:12:05.199771 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:05.199726    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:12:10.204414 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:10.204381    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:12:10.204886 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:10.204856    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:12:17.342235 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:17.342201    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:17.614079 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:17.613993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:20.204928 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:20.204882    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:12:27.541724 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:27.541686    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:27.616811 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:27.616774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:27.682111 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:27.682063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:27.889666 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:27.889586    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:30.204913 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:30.204871    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:12:40.205654 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:40.205608    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:12:47.585890 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.585852    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:47.635875 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.635832    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"]
Sep 09 20:12:47.639823 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.639798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.642738 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.642718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-96f8b-predictor-serving-cert\""
Sep 09 20:12:47.642856 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.642757    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-96f8b-kube-rbac-proxy-sar-config\""
Sep 09 20:12:47.652611 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.652587    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"]
Sep 09 20:12:47.716496 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.716453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f590bc60-f903-4ef4-879a-a4e46ea3c560-kserve-provision-location\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.716496 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.716495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-cabundle-cert\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.716715 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.716615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrkd\" (UniqueName: \"kubernetes.io/projected/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-api-access-cmrkd\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.716715 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.716681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.716858 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.716806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.818425 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.818333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.818425 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.818420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f590bc60-f903-4ef4-879a-a4e46ea3c560-kserve-provision-location\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.818641 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.818450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-cabundle-cert\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.818641 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.818496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrkd\" (UniqueName: \"kubernetes.io/projected/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-api-access-cmrkd\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.818641 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.818561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.818791 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:12:47.818703    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-serving-cert: secret "isvc-logger-raw-96f8b-predictor-serving-cert" not found
Sep 09 20:12:47.818791 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:12:47.818778    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls podName:f590bc60-f903-4ef4-879a-a4e46ea3c560 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:48.318756261 +0000 UTC m=+1056.346287230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls") pod "isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" (UID: "f590bc60-f903-4ef4-879a-a4e46ea3c560") : secret "isvc-logger-raw-96f8b-predictor-serving-cert" not found
Sep 09 20:12:47.818897 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.818864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f590bc60-f903-4ef4-879a-a4e46ea3c560-kserve-provision-location\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.819165 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.819141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.819204 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.819141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-cabundle-cert\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:47.828724 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:47.828691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrkd\" (UniqueName: \"kubernetes.io/projected/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-api-access-cmrkd\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:48.116158 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:48.116063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:48.323944 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:48.323906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:48.326646 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:48.326624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls\") pod \"isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") " pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:48.553343 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:48.553305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:48.834227 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:48.834200    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"]
Sep 09 20:12:48.836437 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:12:48.836409    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf590bc60_f903_4ef4_879a_a4e46ea3c560.slice/crio-95f683e2205b71426232729d879042cfdd3f67945c041b33740b685071c13da2 WatchSource:0}: Error finding container 95f683e2205b71426232729d879042cfdd3f67945c041b33740b685071c13da2: Status 404 returned error can't find the container with id 95f683e2205b71426232729d879042cfdd3f67945c041b33740b685071c13da2
Sep 09 20:12:49.420093 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:49.420055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerStarted","Data":"8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70"}
Sep 09 20:12:49.420263 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:49.420128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerStarted","Data":"95f683e2205b71426232729d879042cfdd3f67945c041b33740b685071c13da2"}
Sep 09 20:12:50.205297 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:50.205255    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:12:53.443522 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:53.443484    2566 generic.go:356] "Generic (PLEG): container finished" podID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerID="8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70" exitCode=0
Sep 09 20:12:53.443963 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:53.443546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerDied","Data":"8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70"}
Sep 09 20:12:54.451176 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:54.451140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerStarted","Data":"ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec"}
Sep 09 20:12:54.451580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:54.451208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerStarted","Data":"039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123"}
Sep 09 20:12:54.451580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:54.451236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerStarted","Data":"21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9"}
Sep 09 20:12:54.451580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:54.451544    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:54.451682 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:54.451670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:54.453119 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:54.453089    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:12:54.474457 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:54.474407    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podStartSLOduration=3.907702344 podStartE2EDuration="7.474389105s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.566686761s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:54.473710092 +0000 UTC m=+1062.501241078" watchObservedRunningTime="2026-09-09 20:12:54.474389105 +0000 UTC m=+1062.501920088"
Sep 09 20:12:55.455892 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:55.455851    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:12:55.456395 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:55.456048    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:12:55.457308 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:55.457264    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:56.460809 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:56.460764    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:12:56.461405 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:12:56.461376    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:13:00.205051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:00.205010    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:13:01.465699 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:01.465669    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:13:01.466228 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:01.466197    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:13:01.466707 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:01.466683    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:13:10.204887 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:10.204793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:13:11.466240 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:11.466196    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:13:11.466724 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:11.466697    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:13:17.448669 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:17.448621    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:17.547553 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:17.547491    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:20.206076 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:20.206040    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:13:21.466382 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:21.466321    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:13:21.466875 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:21.466839    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:13:27.384462 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:27.384418    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:31.466257 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:31.466208    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:13:31.466727 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:31.466678    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:13:41.466711 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:41.466664    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:13:41.467235 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:41.467081    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:13:42.522650 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:42.522610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:43.013200 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:43.013163    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:51.466893 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:51.466844    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:13:51.467435 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:51.467241    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:13:58.602639 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:58.602537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:58.611818 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:58.611781    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"]
Sep 09 20:13:58.612084 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:58.612056    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" containerID="cri-o://b0fecb3f97a787fcae1466b9b0984c7eb6d7659c3ad812560cc3d185588de783" gracePeriod=30
Sep 09 20:13:58.612165 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:58.612131    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kube-rbac-proxy" containerID="cri-o://1835c38d2a3ee8c48328cf43b735c73f80cc5c85011e6d7d89bdd30ffbb9c8a7" gracePeriod=30
Sep 09 20:13:58.767745 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:58.767707    2566 generic.go:356] "Generic (PLEG): container finished" podID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerID="1835c38d2a3ee8c48328cf43b735c73f80cc5c85011e6d7d89bdd30ffbb9c8a7" exitCode=2
Sep 09 20:13:58.767949 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:58.767776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerDied","Data":"1835c38d2a3ee8c48328cf43b735c73f80cc5c85011e6d7d89bdd30ffbb9c8a7"}
Sep 09 20:13:59.921242 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:13:59.921199    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:00.200306 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:00.200218    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:14:00.205625 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:00.205596    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:14:01.466432 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:01.466381    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:14:01.466924 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:01.466902    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:14:02.467033 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:02.466989    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:14:02.467498 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:02.467295    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:14:03.792149 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.792113    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:03.798754 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.798725    2566 generic.go:356] "Generic (PLEG): container finished" podID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerID="b0fecb3f97a787fcae1466b9b0984c7eb6d7659c3ad812560cc3d185588de783" exitCode=0
Sep 09 20:14:03.798906 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.798839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerDied","Data":"b0fecb3f97a787fcae1466b9b0984c7eb6d7659c3ad812560cc3d185588de783"}
Sep 09 20:14:03.872257 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.872232    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:14:03.916662 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.916622    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ac1af84-12bb-48bf-84d3-6d241477127a-kserve-provision-location\") pod \"1ac1af84-12bb-48bf-84d3-6d241477127a\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") "
Sep 09 20:14:03.916662 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.916657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ac1af84-12bb-48bf-84d3-6d241477127a-proxy-tls\") pod \"1ac1af84-12bb-48bf-84d3-6d241477127a\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") "
Sep 09 20:14:03.916936 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.916699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-rbac-proxy-sar-config\") pod \"1ac1af84-12bb-48bf-84d3-6d241477127a\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") "
Sep 09 20:14:03.916936 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.916730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgls6\" (UniqueName: \"kubernetes.io/projected/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-api-access-fgls6\") pod \"1ac1af84-12bb-48bf-84d3-6d241477127a\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") "
Sep 09 20:14:03.916936 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.916803    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-cabundle-cert\") pod \"1ac1af84-12bb-48bf-84d3-6d241477127a\" (UID: \"1ac1af84-12bb-48bf-84d3-6d241477127a\") "
Sep 09 20:14:03.917091 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.917037    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac1af84-12bb-48bf-84d3-6d241477127a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ac1af84-12bb-48bf-84d3-6d241477127a" (UID: "1ac1af84-12bb-48bf-84d3-6d241477127a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:03.917148 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.917109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ac1af84-12bb-48bf-84d3-6d241477127a" (UID: "1ac1af84-12bb-48bf-84d3-6d241477127a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:03.917148 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.917133    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ac1af84-12bb-48bf-84d3-6d241477127a-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:03.917407 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.917354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ac1af84-12bb-48bf-84d3-6d241477127a" (UID: "1ac1af84-12bb-48bf-84d3-6d241477127a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:03.919131 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.919101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac1af84-12bb-48bf-84d3-6d241477127a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ac1af84-12bb-48bf-84d3-6d241477127a" (UID: "1ac1af84-12bb-48bf-84d3-6d241477127a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:03.919345 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.919323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-api-access-fgls6" (OuterVolumeSpecName: "kube-api-access-fgls6") pod "1ac1af84-12bb-48bf-84d3-6d241477127a" (UID: "1ac1af84-12bb-48bf-84d3-6d241477127a"). InnerVolumeSpecName "kube-api-access-fgls6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:03.975826 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:03.975794    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:04.018617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.018579    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:04.018617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.018610    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgls6\" (UniqueName: \"kubernetes.io/projected/1ac1af84-12bb-48bf-84d3-6d241477127a-kube-api-access-fgls6\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:04.018617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.018621    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ac1af84-12bb-48bf-84d3-6d241477127a-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:04.018617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.018629    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ac1af84-12bb-48bf-84d3-6d241477127a-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:04.805646 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.805555    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"
Sep 09 20:14:04.805646 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.805567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j" event={"ID":"1ac1af84-12bb-48bf-84d3-6d241477127a","Type":"ContainerDied","Data":"71eae3eecbb2a47fb286308d03ea8c204e0feff8cb6f4c454a4bb43c431d732d"}
Sep 09 20:14:04.805646 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.805629    2566 scope.go:117] "RemoveContainer" containerID="1835c38d2a3ee8c48328cf43b735c73f80cc5c85011e6d7d89bdd30ffbb9c8a7"
Sep 09 20:14:04.814581 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.814559    2566 scope.go:117] "RemoveContainer" containerID="b0fecb3f97a787fcae1466b9b0984c7eb6d7659c3ad812560cc3d185588de783"
Sep 09 20:14:04.822786 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.822763    2566 scope.go:117] "RemoveContainer" containerID="31baf446cd7af3991e8b1baf0fbe86292d768fcfd297171df3364b172d707190"
Sep 09 20:14:04.827313 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.827288    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"]
Sep 09 20:14:04.831482 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:04.831459    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-05a1b8-predictor-7cf88d7b7d-2qz2j"]
Sep 09 20:14:06.470505 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:06.470470    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" path="/var/lib/kubelet/pods/1ac1af84-12bb-48bf-84d3-6d241477127a/volumes"
Sep 09 20:14:08.829714 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:08.829677    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:10.667343 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:10.667304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:12.470755 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:12.470727    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:14:12.471230 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:12.470815    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:14:14.020838 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:14.020803    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:14.215640 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:14.215603    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:22.816131 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:22.816098    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"]
Sep 09 20:14:22.816677 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:22.816399    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy" containerID="cri-o://039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123" gracePeriod=30
Sep 09 20:14:22.816677 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:22.816415    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" containerID="cri-o://ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec" gracePeriod=30
Sep 09 20:14:22.816677 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:22.816354    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" containerID="cri-o://21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9" gracePeriod=30
Sep 09 20:14:22.837528 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:22.837500    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:22.934209 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:22.934172    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:23.040175 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:23.040134    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:23.334948 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:23.334915    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:23.898252 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:23.898216    2566 generic.go:356] "Generic (PLEG): container finished" podID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerID="039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123" exitCode=2
Sep 09 20:14:23.898648 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:23.898278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerDied","Data":"039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123"}
Sep 09 20:14:26.462034 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:26.461994    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:14:27.919747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:27.919664    2566 generic.go:356] "Generic (PLEG): container finished" podID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerID="21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9" exitCode=0
Sep 09 20:14:27.919747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:27.919734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerDied","Data":"21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9"}
Sep 09 20:14:31.461178 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:31.461130    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:14:32.467542 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:32.467496    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:14:32.467983 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:32.467873    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:14:36.461558 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:36.461461    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:14:36.461949 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:36.461636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:14:41.461464 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:41.461417    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:14:42.466940 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:42.466899    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:14:42.467458 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:42.467239    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:14:46.461174 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:46.461118    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:14:51.461250 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:51.461190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:14:52.466971 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:52.466910    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:14:52.467514 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:52.467222    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:14:52.470609 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:52.470586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:14:52.470740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:52.470692    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:14:52.974503 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:52.974481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:14:53.039908 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.039872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-cabundle-cert\") pod \"f590bc60-f903-4ef4-879a-a4e46ea3c560\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") "
Sep 09 20:14:53.040123 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.039944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-rbac-proxy-sar-config\") pod \"f590bc60-f903-4ef4-879a-a4e46ea3c560\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") "
Sep 09 20:14:53.040123 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.039981    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f590bc60-f903-4ef4-879a-a4e46ea3c560-kserve-provision-location\") pod \"f590bc60-f903-4ef4-879a-a4e46ea3c560\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") "
Sep 09 20:14:53.040123 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.040092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmrkd\" (UniqueName: \"kubernetes.io/projected/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-api-access-cmrkd\") pod \"f590bc60-f903-4ef4-879a-a4e46ea3c560\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") "
Sep 09 20:14:53.040123 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.040120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls\") pod \"f590bc60-f903-4ef4-879a-a4e46ea3c560\" (UID: \"f590bc60-f903-4ef4-879a-a4e46ea3c560\") "
Sep 09 20:14:53.040425 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.040401    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f590bc60-f903-4ef4-879a-a4e46ea3c560" (UID: "f590bc60-f903-4ef4-879a-a4e46ea3c560"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:53.040425 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.040408    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f590bc60-f903-4ef4-879a-a4e46ea3c560-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f590bc60-f903-4ef4-879a-a4e46ea3c560" (UID: "f590bc60-f903-4ef4-879a-a4e46ea3c560"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:53.040552 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.040514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f590bc60-f903-4ef4-879a-a4e46ea3c560" (UID: "f590bc60-f903-4ef4-879a-a4e46ea3c560"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:53.042540 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.042511    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-api-access-cmrkd" (OuterVolumeSpecName: "kube-api-access-cmrkd") pod "f590bc60-f903-4ef4-879a-a4e46ea3c560" (UID: "f590bc60-f903-4ef4-879a-a4e46ea3c560"). InnerVolumeSpecName "kube-api-access-cmrkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:53.042742 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.042724    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f590bc60-f903-4ef4-879a-a4e46ea3c560" (UID: "f590bc60-f903-4ef4-879a-a4e46ea3c560"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:53.047345 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.047314    2566 generic.go:356] "Generic (PLEG): container finished" podID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerID="ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec" exitCode=0
Sep 09 20:14:53.047467 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.047388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerDied","Data":"ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec"}
Sep 09 20:14:53.047467 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.047403    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"
Sep 09 20:14:53.047467 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.047455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p" event={"ID":"f590bc60-f903-4ef4-879a-a4e46ea3c560","Type":"ContainerDied","Data":"95f683e2205b71426232729d879042cfdd3f67945c041b33740b685071c13da2"}
Sep 09 20:14:53.047589 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.047491    2566 scope.go:117] "RemoveContainer" containerID="ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec"
Sep 09 20:14:53.059672 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.059655    2566 scope.go:117] "RemoveContainer" containerID="039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123"
Sep 09 20:14:53.067596 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.067578    2566 scope.go:117] "RemoveContainer" containerID="21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9"
Sep 09 20:14:53.073103 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.073052    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"]
Sep 09 20:14:53.076328 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.076310    2566 scope.go:117] "RemoveContainer" containerID="8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70"
Sep 09 20:14:53.077407 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.077355    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-96f8b-predictor-59d789d6dc-5d48p"]
Sep 09 20:14:53.084304 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.084282    2566 scope.go:117] "RemoveContainer" containerID="ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec"
Sep 09 20:14:53.084629 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:14:53.084610    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec\": container with ID starting with ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec not found: ID does not exist" containerID="ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec"
Sep 09 20:14:53.084710 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.084645    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec"} err="failed to get container status \"ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec\": rpc error: code = NotFound desc = could not find container \"ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec\": container with ID starting with ff463ad2b5b8efed7115d4226e63f142fbaadf3beb315ac6244d7c319ad3c6ec not found: ID does not exist"
Sep 09 20:14:53.084710 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.084671    2566 scope.go:117] "RemoveContainer" containerID="039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123"
Sep 09 20:14:53.085001 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:14:53.084982    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123\": container with ID starting with 039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123 not found: ID does not exist" containerID="039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123"
Sep 09 20:14:53.085054 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.085008    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123"} err="failed to get container status \"039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123\": rpc error: code = NotFound desc = could not find container \"039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123\": container with ID starting with 039a748c35155c27ec68300d097afb39ca74566e11d7bc34372e8574c7615123 not found: ID does not exist"
Sep 09 20:14:53.085054 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.085026    2566 scope.go:117] "RemoveContainer" containerID="21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9"
Sep 09 20:14:53.085278 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:14:53.085260    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9\": container with ID starting with 21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9 not found: ID does not exist" containerID="21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9"
Sep 09 20:14:53.085341 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.085287    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9"} err="failed to get container status \"21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9\": rpc error: code = NotFound desc = could not find container \"21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9\": container with ID starting with 21473181c0a11de5edd52344d81823514607dc4e03486b4b385ce38b4449dda9 not found: ID does not exist"
Sep 09 20:14:53.085341 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.085308    2566 scope.go:117] "RemoveContainer" containerID="8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70"
Sep 09 20:14:53.085575 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:14:53.085558    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70\": container with ID starting with 8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70 not found: ID does not exist" containerID="8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70"
Sep 09 20:14:53.085622 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.085582    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70"} err="failed to get container status \"8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70\": rpc error: code = NotFound desc = could not find container \"8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70\": container with ID starting with 8d78f7f72f76063ac745afddd337e9f9086f951b9c52fec920393a04f44e3d70 not found: ID does not exist"
Sep 09 20:14:53.141252 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.141211    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f590bc60-f903-4ef4-879a-a4e46ea3c560-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.141252 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.141244    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmrkd\" (UniqueName: \"kubernetes.io/projected/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-api-access-cmrkd\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.141252 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.141256    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f590bc60-f903-4ef4-879a-a4e46ea3c560-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.141532 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.141265    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.141532 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:53.141274    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f590bc60-f903-4ef4-879a-a4e46ea3c560-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:14:54.471047 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:14:54.471016    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" path="/var/lib/kubelet/pods/f590bc60-f903-4ef4-879a-a4e46ea3c560/volumes"
Sep 09 20:15:12.500687 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:12.500659    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:15:12.503239 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:12.503214    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:15:33.886853 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:33.886815    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:44.136987 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:44.136925    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:49.288482 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.288443    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:49.318138 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318101    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"]
Sep 09 20:15:49.318509 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318495    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container"
Sep 09 20:15:49.318557 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318511    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container"
Sep 09 20:15:49.318557 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318521    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy"
Sep 09 20:15:49.318557 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318526    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy"
Sep 09 20:15:49.318557 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318535    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="storage-initializer"
Sep 09 20:15:49.318557 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318541    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="storage-initializer"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318567    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="storage-initializer"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318572    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="storage-initializer"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318580    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kube-rbac-proxy"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318585    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kube-rbac-proxy"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318593    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318598    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318605    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318610    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318667    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kube-rbac-proxy"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318677    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kserve-container"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318684    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="kube-rbac-proxy"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318692    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac1af84-12bb-48bf-84d3-6d241477127a" containerName="kserve-container"
Sep 09 20:15:49.318750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.318698    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f590bc60-f903-4ef4-879a-a4e46ea3c560" containerName="agent"
Sep 09 20:15:49.322015 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.321998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.325120 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.325083    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-be712-predictor-serving-cert\""
Sep 09 20:15:49.325120 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.325093    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:15:49.325120 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.325099    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:15:49.325384 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.325136    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:15:49.325384 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.325161    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-be712-kube-rbac-proxy-sar-config\""
Sep 09 20:15:49.326416 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.326347    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:15:49.326416 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.326353    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:15:49.332435 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.332400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"]
Sep 09 20:15:49.428576 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.428536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.428780 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.428582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.428780 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.428645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnlw\" (UniqueName: \"kubernetes.io/projected/eafcb965-e16f-4543-adb8-453984c52278-kube-api-access-7vnlw\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.428780 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.428725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-cabundle-cert\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.428780 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.428757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafcb965-e16f-4543-adb8-453984c52278-kserve-provision-location\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529175 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529285 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529285 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnlw\" (UniqueName: \"kubernetes.io/projected/eafcb965-e16f-4543-adb8-453984c52278-kube-api-access-7vnlw\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529285 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-cabundle-cert\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529445 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:15:49.529304    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-serving-cert: secret "raw-sklearn-runtime-be712-predictor-serving-cert" not found
Sep 09 20:15:49.529445 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafcb965-e16f-4543-adb8-453984c52278-kserve-provision-location\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529543 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:15:49.529478    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls podName:eafcb965-e16f-4543-adb8-453984c52278 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:50.02938744 +0000 UTC m=+1238.056918419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls") pod "raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" (UID: "eafcb965-e16f-4543-adb8-453984c52278") : secret "raw-sklearn-runtime-be712-predictor-serving-cert" not found
Sep 09 20:15:49.529704 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafcb965-e16f-4543-adb8-453984c52278-kserve-provision-location\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529935 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.529983 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.529946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-cabundle-cert\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.541676 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.541620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnlw\" (UniqueName: \"kubernetes.io/projected/eafcb965-e16f-4543-adb8-453984c52278-kube-api-access-7vnlw\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:49.575713 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:49.575680    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:50.032976 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:50.032922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:50.035638 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:50.035614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls\") pod \"raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:50.234064 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:50.234017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:50.509805 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:50.509776    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"]
Sep 09 20:15:50.512112 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:15:50.512079    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafcb965_e16f_4543_adb8_453984c52278.slice/crio-bb3937e8e689aceeb91454defe46a40ddf5e2e05b14f8a767fa72d8ea2ca9efa WatchSource:0}: Error finding container bb3937e8e689aceeb91454defe46a40ddf5e2e05b14f8a767fa72d8ea2ca9efa: Status 404 returned error can't find the container with id bb3937e8e689aceeb91454defe46a40ddf5e2e05b14f8a767fa72d8ea2ca9efa
Sep 09 20:15:51.320946 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:51.320906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerStarted","Data":"fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0"}
Sep 09 20:15:51.321134 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:51.320975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerStarted","Data":"bb3937e8e689aceeb91454defe46a40ddf5e2e05b14f8a767fa72d8ea2ca9efa"}
Sep 09 20:15:55.342136 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:55.342101    2566 generic.go:356] "Generic (PLEG): container finished" podID="eafcb965-e16f-4543-adb8-453984c52278" containerID="fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0" exitCode=0
Sep 09 20:15:55.342535 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:55.342178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerDied","Data":"fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0"}
Sep 09 20:15:56.349807 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:56.349770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerStarted","Data":"fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39"}
Sep 09 20:15:56.350196 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:56.349881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerStarted","Data":"dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404"}
Sep 09 20:15:56.350248 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:56.350213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:56.350403 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:56.350352    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:15:56.351637 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:56.351608    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:15:56.369485 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:56.369421    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podStartSLOduration=3.586141535 podStartE2EDuration="7.369397968s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783256433s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:56.36814458 +0000 UTC m=+1244.395675570" watchObservedRunningTime="2026-09-09 20:15:56.369397968 +0000 UTC m=+1244.396928954"
Sep 09 20:15:57.354865 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:15:57.354831    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:16:02.359944 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:02.359914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:16:02.360468 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:02.360442    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:16:12.361302 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:12.361212    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:16:22.361172 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:22.361130    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:16:25.580707 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:25.580670    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:25.638303 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:25.638272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:32.361381 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:32.361322    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:16:42.360774 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:42.360733    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:16:52.360523 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:16:52.360477    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:17:02.360437 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:02.360391    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:17:04.464961 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:04.464907    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:17:13.046098 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:13.046047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:14.470465 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:14.470438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:17:18.208814 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.208774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:18.226090 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.226046    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"]
Sep 09 20:17:18.229762 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.229738    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.232612 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.232587    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:17:18.232747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.232595    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:17:18.239678 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.239650    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"]
Sep 09 20:17:18.309774 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.309725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lw9\" (UniqueName: \"kubernetes.io/projected/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-api-access-c6lw9\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.309990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.309797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.309990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.309883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.309990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.309915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.309990 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.309939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.310880 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.310853    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:18.410883 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.410847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lw9\" (UniqueName: \"kubernetes.io/projected/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-api-access-c6lw9\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.411188 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.410903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.411188 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.410959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.411188 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.410976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.411188 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.410992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.411188 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:17:18.411118    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 20:17:18.411465 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:17:18.411207    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls podName:dff205a2-c12d-44ea-bd3a-dc7b2aa53843 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:18.911184986 +0000 UTC m=+1326.938715953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls") pod "isvc-canary-create-predictor-68984b9498-2dpcc" (UID: "dff205a2-c12d-44ea-bd3a-dc7b2aa53843") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 20:17:18.411560 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.411535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.411799 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.411779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.411836 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.411790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.422047 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.422013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lw9\" (UniqueName: \"kubernetes.io/projected/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-api-access-c6lw9\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.425413 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.425387    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:18.564753 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.564721    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:18.915472 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.915351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:18.917983 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:18.917948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-2dpcc\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:19.142700 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.142657    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:19.379497 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.379462    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"]
Sep 09 20:17:19.379983 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.379731    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" containerID="cri-o://dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404" gracePeriod=30
Sep 09 20:17:19.379983 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.379775    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kube-rbac-proxy" containerID="cri-o://fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39" gracePeriod=30
Sep 09 20:17:19.418638 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.418602    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"]
Sep 09 20:17:19.421935 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:17:19.421907    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff205a2_c12d_44ea_bd3a_dc7b2aa53843.slice/crio-d3bd176ba0b28d46ac45863c991bbe0b510e0d833a452d20bd9b0c33387bd027 WatchSource:0}: Error finding container d3bd176ba0b28d46ac45863c991bbe0b510e0d833a452d20bd9b0c33387bd027: Status 404 returned error can't find the container with id d3bd176ba0b28d46ac45863c991bbe0b510e0d833a452d20bd9b0c33387bd027
Sep 09 20:17:19.453670 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.453568    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:19.752224 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.752185    2566 generic.go:356] "Generic (PLEG): container finished" podID="eafcb965-e16f-4543-adb8-453984c52278" containerID="fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39" exitCode=2
Sep 09 20:17:19.752425 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.752257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerDied","Data":"fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39"}
Sep 09 20:17:19.754116 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.754088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerStarted","Data":"9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a"}
Sep 09 20:17:19.754273 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:19.754141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerStarted","Data":"d3bd176ba0b28d46ac45863c991bbe0b510e0d833a452d20bd9b0c33387bd027"}
Sep 09 20:17:22.355413 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:22.355342    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 20:17:23.776050 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:23.776017    2566 generic.go:356] "Generic (PLEG): container finished" podID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerID="9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a" exitCode=0
Sep 09 20:17:23.776463 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:23.776069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerDied","Data":"9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a"}
Sep 09 20:17:24.531164 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.531133    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:17:24.571582 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571509    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-cabundle-cert\") pod \"eafcb965-e16f-4543-adb8-453984c52278\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") "
Sep 09 20:17:24.571582 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571561    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafcb965-e16f-4543-adb8-453984c52278-kserve-provision-location\") pod \"eafcb965-e16f-4543-adb8-453984c52278\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") "
Sep 09 20:17:24.571805 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571584    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-kube-rbac-proxy-sar-config\") pod \"eafcb965-e16f-4543-adb8-453984c52278\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") "
Sep 09 20:17:24.571805 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vnlw\" (UniqueName: \"kubernetes.io/projected/eafcb965-e16f-4543-adb8-453984c52278-kube-api-access-7vnlw\") pod \"eafcb965-e16f-4543-adb8-453984c52278\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") "
Sep 09 20:17:24.571805 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls\") pod \"eafcb965-e16f-4543-adb8-453984c52278\" (UID: \"eafcb965-e16f-4543-adb8-453984c52278\") "
Sep 09 20:17:24.571971 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571882    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafcb965-e16f-4543-adb8-453984c52278-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eafcb965-e16f-4543-adb8-453984c52278" (UID: "eafcb965-e16f-4543-adb8-453984c52278"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:17:24.571971 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571930    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eafcb965-e16f-4543-adb8-453984c52278" (UID: "eafcb965-e16f-4543-adb8-453984c52278"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:24.572078 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.571987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eafcb965-e16f-4543-adb8-453984c52278" (UID: "eafcb965-e16f-4543-adb8-453984c52278"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:24.572078 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.572012    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:17:24.572078 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.572032    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafcb965-e16f-4543-adb8-453984c52278-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:17:24.574166 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.574138    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eafcb965-e16f-4543-adb8-453984c52278" (UID: "eafcb965-e16f-4543-adb8-453984c52278"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:24.574278 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.574169    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafcb965-e16f-4543-adb8-453984c52278-kube-api-access-7vnlw" (OuterVolumeSpecName: "kube-api-access-7vnlw") pod "eafcb965-e16f-4543-adb8-453984c52278" (UID: "eafcb965-e16f-4543-adb8-453984c52278"). InnerVolumeSpecName "kube-api-access-7vnlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:24.673544 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.673513    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafcb965-e16f-4543-adb8-453984c52278-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:17:24.673544 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.673540    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnlw\" (UniqueName: \"kubernetes.io/projected/eafcb965-e16f-4543-adb8-453984c52278-kube-api-access-7vnlw\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:17:24.673544 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.673552    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafcb965-e16f-4543-adb8-453984c52278-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:17:24.783252 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.783217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerStarted","Data":"6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400"}
Sep 09 20:17:24.783718 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.783297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerStarted","Data":"734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87"}
Sep 09 20:17:24.783718 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.783544    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:24.785459 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.785425    2566 generic.go:356] "Generic (PLEG): container finished" podID="eafcb965-e16f-4543-adb8-453984c52278" containerID="dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404" exitCode=0
Sep 09 20:17:24.785580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.785478    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"
Sep 09 20:17:24.785580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.785500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerDied","Data":"dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404"}
Sep 09 20:17:24.785580 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.785566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" event={"ID":"eafcb965-e16f-4543-adb8-453984c52278","Type":"ContainerDied","Data":"bb3937e8e689aceeb91454defe46a40ddf5e2e05b14f8a767fa72d8ea2ca9efa"}
Sep 09 20:17:24.785699 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.785608    2566 scope.go:117] "RemoveContainer" containerID="fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39"
Sep 09 20:17:24.794407 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.794384    2566 scope.go:117] "RemoveContainer" containerID="dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404"
Sep 09 20:17:24.802230 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.802209    2566 scope.go:117] "RemoveContainer" containerID="fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0"
Sep 09 20:17:24.804567 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.804528    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podStartSLOduration=3.267320124 podStartE2EDuration="6.804513067s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.537192943s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:24.803035307 +0000 UTC m=+1332.830566356" watchObservedRunningTime="2026-09-09 20:17:24.804513067 +0000 UTC m=+1332.832044115"
Sep 09 20:17:24.816583 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.816564    2566 scope.go:117] "RemoveContainer" containerID="fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39"
Sep 09 20:17:24.816916 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:17:24.816895    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39\": container with ID starting with fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39 not found: ID does not exist" containerID="fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39"
Sep 09 20:17:24.817009 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.816923    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39"} err="failed to get container status \"fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39\": rpc error: code = NotFound desc = could not find container \"fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39\": container with ID starting with fdf6f27a96ac73e350e41d116620a87fcbdeb5c012a23a4d8f97be37e9fa2a39 not found: ID does not exist"
Sep 09 20:17:24.817009 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.816951    2566 scope.go:117] "RemoveContainer" containerID="dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404"
Sep 09 20:17:24.817224 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:17:24.817209    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404\": container with ID starting with dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404 not found: ID does not exist" containerID="dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404"
Sep 09 20:17:24.817270 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.817228    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404"} err="failed to get container status \"dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404\": rpc error: code = NotFound desc = could not find container \"dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404\": container with ID starting with dfafda30e0358e1acc504509df426d4973f24c0c0836ba40db679d6b8ae03404 not found: ID does not exist"
Sep 09 20:17:24.817270 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.817240    2566 scope.go:117] "RemoveContainer" containerID="fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0"
Sep 09 20:17:24.817477 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:17:24.817459    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0\": container with ID starting with fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0 not found: ID does not exist" containerID="fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0"
Sep 09 20:17:24.817617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.817481    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0"} err="failed to get container status \"fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0\": rpc error: code = NotFound desc = could not find container \"fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0\": container with ID starting with fa326eeda0fe8ddfb813f956ff079cdb3704c252c2ccb20a4fda2477191f42c0 not found: ID does not exist"
Sep 09 20:17:24.824207 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.824157    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"]
Sep 09 20:17:24.827512 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:24.827474    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm"]
Sep 09 20:17:25.465330 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:25.465283    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-be712-predictor-5cf9f77dc-xswdm" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: i/o timeout"
Sep 09 20:17:25.792074 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:25.792045    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:25.793298 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:25.793272    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:17:26.470654 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:26.470622    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eafcb965-e16f-4543-adb8-453984c52278" path="/var/lib/kubelet/pods/eafcb965-e16f-4543-adb8-453984c52278/volumes"
Sep 09 20:17:26.796579 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:26.796485    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:17:31.801276 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:31.801246    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:17:31.801776 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:31.801745    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:17:41.801947 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:41.801860    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:17:51.802061 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:17:51.802019    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:18:01.801922 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:01.801883    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:18:11.801873 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:11.801833    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:18:21.801915 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:21.801878    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:18:31.802048 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:31.802003    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:18:40.842991 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:40.842950    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"]
Sep 09 20:18:40.843497 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:40.843188    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container" containerID="cri-o://734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87" gracePeriod=30
Sep 09 20:18:40.843497 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:40.843260    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kube-rbac-proxy" containerID="cri-o://6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400" gracePeriod=30
Sep 09 20:18:40.844520 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:40.844498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:18:41.149405 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:41.149304    2566 generic.go:356] "Generic (PLEG): container finished" podID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerID="6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400" exitCode=2
Sep 09 20:18:41.149405 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:41.149386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerDied","Data":"6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400"}
Sep 09 20:18:41.797283 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:41.797236    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 09 20:18:45.991613 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:45.991589    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:18:46.116161 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116074    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-rbac-proxy-sar-config\") pod \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") "
Sep 09 20:18:46.116161 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-cabundle-cert\") pod \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") "
Sep 09 20:18:46.116417 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116218    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6lw9\" (UniqueName: \"kubernetes.io/projected/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-api-access-c6lw9\") pod \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") "
Sep 09 20:18:46.116417 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116243    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kserve-provision-location\") pod \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") "
Sep 09 20:18:46.116417 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls\") pod \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\" (UID: \"dff205a2-c12d-44ea-bd3a-dc7b2aa53843\") "
Sep 09 20:18:46.116583 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dff205a2-c12d-44ea-bd3a-dc7b2aa53843" (UID: "dff205a2-c12d-44ea-bd3a-dc7b2aa53843"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:46.116583 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116556    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dff205a2-c12d-44ea-bd3a-dc7b2aa53843" (UID: "dff205a2-c12d-44ea-bd3a-dc7b2aa53843"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:46.116672 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.116597    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dff205a2-c12d-44ea-bd3a-dc7b2aa53843" (UID: "dff205a2-c12d-44ea-bd3a-dc7b2aa53843"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:18:46.118500 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.118479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dff205a2-c12d-44ea-bd3a-dc7b2aa53843" (UID: "dff205a2-c12d-44ea-bd3a-dc7b2aa53843"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:46.118589 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.118566    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-api-access-c6lw9" (OuterVolumeSpecName: "kube-api-access-c6lw9") pod "dff205a2-c12d-44ea-bd3a-dc7b2aa53843" (UID: "dff205a2-c12d-44ea-bd3a-dc7b2aa53843"). InnerVolumeSpecName "kube-api-access-c6lw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:46.175133 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.175093    2566 generic.go:356] "Generic (PLEG): container finished" podID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerID="734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87" exitCode=0
Sep 09 20:18:46.175324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.175158    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"
Sep 09 20:18:46.175324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.175184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerDied","Data":"734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87"}
Sep 09 20:18:46.175324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.175255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc" event={"ID":"dff205a2-c12d-44ea-bd3a-dc7b2aa53843","Type":"ContainerDied","Data":"d3bd176ba0b28d46ac45863c991bbe0b510e0d833a452d20bd9b0c33387bd027"}
Sep 09 20:18:46.175324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.175285    2566 scope.go:117] "RemoveContainer" containerID="6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400"
Sep 09 20:18:46.190891 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.190868    2566 scope.go:117] "RemoveContainer" containerID="734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87"
Sep 09 20:18:46.198814 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.198788    2566 scope.go:117] "RemoveContainer" containerID="9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a"
Sep 09 20:18:46.203869 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.203842    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"]
Sep 09 20:18:46.207407 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.207382    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-2dpcc"]
Sep 09 20:18:46.207978 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.207962    2566 scope.go:117] "RemoveContainer" containerID="6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400"
Sep 09 20:18:46.208288 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:18:46.208270    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400\": container with ID starting with 6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400 not found: ID does not exist" containerID="6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400"
Sep 09 20:18:46.208393 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.208297    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400"} err="failed to get container status \"6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400\": rpc error: code = NotFound desc = could not find container \"6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400\": container with ID starting with 6f485facc13ba411245c9de3b26d2f1b864c6f8cc7c2ffaaa848acd8f615b400 not found: ID does not exist"
Sep 09 20:18:46.208393 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.208317    2566 scope.go:117] "RemoveContainer" containerID="734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87"
Sep 09 20:18:46.208634 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:18:46.208609    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87\": container with ID starting with 734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87 not found: ID does not exist" containerID="734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87"
Sep 09 20:18:46.208727 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.208642    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87"} err="failed to get container status \"734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87\": rpc error: code = NotFound desc = could not find container \"734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87\": container with ID starting with 734dfc47d1b7c680c2380ae239b58a2236b7708cf6336087acdbd36286616e87 not found: ID does not exist"
Sep 09 20:18:46.208727 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.208660    2566 scope.go:117] "RemoveContainer" containerID="9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a"
Sep 09 20:18:46.208920 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:18:46.208903    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a\": container with ID starting with 9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a not found: ID does not exist" containerID="9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a"
Sep 09 20:18:46.208971 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.208934    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a"} err="failed to get container status \"9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a\": rpc error: code = NotFound desc = could not find container \"9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a\": container with ID starting with 9f738ee2348406795b0eac39ad9d19aeffd77fa6dd757ae4c95fe3969040b49a not found: ID does not exist"
Sep 09 20:18:46.217354 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.217334    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:18:46.217444 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.217379    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:18:46.217444 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.217392    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lw9\" (UniqueName: \"kubernetes.io/projected/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kube-api-access-c6lw9\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:18:46.217444 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.217400    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:18:46.217444 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.217411    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff205a2-c12d-44ea-bd3a-dc7b2aa53843-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:18:46.469849 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:46.469820    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" path="/var/lib/kubelet/pods/dff205a2-c12d-44ea-bd3a-dc7b2aa53843/volumes"
Sep 09 20:18:48.312330 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:48.312300    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:48.392028 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:48.391997    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:53.434641 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.434588    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:53.485523 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.485488    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"]
Sep 09 20:18:53.486003 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.485989    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container"
Sep 09 20:18:53.486051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486007    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container"
Sep 09 20:18:53.486051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486022    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kube-rbac-proxy"
Sep 09 20:18:53.486051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486030    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kube-rbac-proxy"
Sep 09 20:18:53.486051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486041    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kube-rbac-proxy"
Sep 09 20:18:53.486051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486049    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kube-rbac-proxy"
Sep 09 20:18:53.486206 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486080    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container"
Sep 09 20:18:53.486206 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486087    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container"
Sep 09 20:18:53.486206 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486097    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="storage-initializer"
Sep 09 20:18:53.486206 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486104    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="storage-initializer"
Sep 09 20:18:53.486206 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486123    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="storage-initializer"
Sep 09 20:18:53.486206 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486131    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="storage-initializer"
Sep 09 20:18:53.486431 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486211    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kserve-container"
Sep 09 20:18:53.486431 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486224    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kserve-container"
Sep 09 20:18:53.486431 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486235    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff205a2-c12d-44ea-bd3a-dc7b2aa53843" containerName="kube-rbac-proxy"
Sep 09 20:18:53.486431 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.486246    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafcb965-e16f-4543-adb8-453984c52278" containerName="kube-rbac-proxy"
Sep 09 20:18:53.491751 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.491724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.494558 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.494530    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:18:53.494755 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.494563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:18:53.494755 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.494578    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:18:53.494934 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.494917    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:18:53.495001 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.494961    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:18:53.495117 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.495003    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:18:53.496297 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.496276    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:18:53.499027 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.499002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"]
Sep 09 20:18:53.519470 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.519437    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:53.578928 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.578880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9f9e93-d81c-47b2-a0da-a0945065220d-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.579148 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.578988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kwn\" (UniqueName: \"kubernetes.io/projected/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-api-access-86kwn\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.579148 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.579034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.579148 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.579064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.579148 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.579109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9f9e93-d81c-47b2-a0da-a0945065220d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.680126 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.680091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kwn\" (UniqueName: \"kubernetes.io/projected/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-api-access-86kwn\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.680322 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.680142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.680322 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.680181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.680322 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.680208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9f9e93-d81c-47b2-a0da-a0945065220d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.680322 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.680239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9f9e93-d81c-47b2-a0da-a0945065220d-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.680882 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.680862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9f9e93-d81c-47b2-a0da-a0945065220d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.681257 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.681221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.681257 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.681235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.683280 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.683256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9f9e93-d81c-47b2-a0da-a0945065220d-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.687833 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.687766    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:53.689068 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.689048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kwn\" (UniqueName: \"kubernetes.io/projected/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-api-access-86kwn\") pod \"isvc-canary-promote-predictor-cb75746f8-drhbg\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.806209 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.806166    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:53.854768 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:53.854735    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:54.087044 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:54.087020    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"]
Sep 09 20:18:54.089500 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:18:54.089470    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9f9e93_d81c_47b2_a0da_a0945065220d.slice/crio-8d407d64aabd939046e671102ee1719ad7479f9fb0a5d484b840e3ae8575474c WatchSource:0}: Error finding container 8d407d64aabd939046e671102ee1719ad7479f9fb0a5d484b840e3ae8575474c: Status 404 returned error can't find the container with id 8d407d64aabd939046e671102ee1719ad7479f9fb0a5d484b840e3ae8575474c
Sep 09 20:18:54.215068 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:54.215025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerStarted","Data":"3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49"}
Sep 09 20:18:54.215264 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:54.215120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerStarted","Data":"8d407d64aabd939046e671102ee1719ad7479f9fb0a5d484b840e3ae8575474c"}
Sep 09 20:18:58.234485 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:58.234451    2566 generic.go:356] "Generic (PLEG): container finished" podID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerID="3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49" exitCode=0
Sep 09 20:18:58.234855 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:58.234514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerDied","Data":"3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49"}
Sep 09 20:18:59.241249 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:59.241215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerStarted","Data":"bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c"}
Sep 09 20:18:59.241674 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:59.241282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerStarted","Data":"a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc"}
Sep 09 20:18:59.241674 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:59.241503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:18:59.260872 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:18:59.260815    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podStartSLOduration=2.5764837800000002 podStartE2EDuration="6.260796794s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684313014s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:59.259022266 +0000 UTC m=+1427.286553255" watchObservedRunningTime="2026-09-09 20:18:59.260796794 +0000 UTC m=+1427.288327826"
Sep 09 20:19:00.246190 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:00.246153    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:19:00.247869 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:00.247843    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:19:01.250882 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:01.250838    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:19:06.255992 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:06.255909    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:19:06.256511 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:06.256433    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:19:16.257300 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:16.257257    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:19:26.256590 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:26.256548    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:19:36.257279 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:36.257235    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:19:46.256615 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:46.256570    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:19:56.256380 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:19:56.256315    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:20:06.257196 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:06.257153    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:20:12.537204 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:12.537162    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:20:12.539382 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:12.539345    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:20:16.249773 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:16.249735    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"]
Sep 09 20:20:16.250184 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:16.250001    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container" containerID="cri-o://a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc" gracePeriod=30
Sep 09 20:20:16.250184 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:16.250056    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kube-rbac-proxy" containerID="cri-o://bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c" gracePeriod=30
Sep 09 20:20:16.251452 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:16.251426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:20:16.255080 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:16.255052    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": EOF"
Sep 09 20:20:16.595087 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:16.594995    2566 generic.go:356] "Generic (PLEG): container finished" podID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerID="bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c" exitCode=2
Sep 09 20:20:16.595087 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:16.595074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerDied","Data":"bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c"}
Sep 09 20:20:21.251582 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.251526    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 20:20:21.404820 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.404794    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:20:21.565666 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.565605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86kwn\" (UniqueName: \"kubernetes.io/projected/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-api-access-86kwn\") pod \"df9f9e93-d81c-47b2-a0da-a0945065220d\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") "
Sep 09 20:20:21.565893 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.565727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9f9e93-d81c-47b2-a0da-a0945065220d-proxy-tls\") pod \"df9f9e93-d81c-47b2-a0da-a0945065220d\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") "
Sep 09 20:20:21.565893 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.565765    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9f9e93-d81c-47b2-a0da-a0945065220d-kserve-provision-location\") pod \"df9f9e93-d81c-47b2-a0da-a0945065220d\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") "
Sep 09 20:20:21.565893 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.565797    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-cabundle-cert\") pod \"df9f9e93-d81c-47b2-a0da-a0945065220d\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") "
Sep 09 20:20:21.565893 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.565839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-rbac-proxy-sar-config\") pod \"df9f9e93-d81c-47b2-a0da-a0945065220d\" (UID: \"df9f9e93-d81c-47b2-a0da-a0945065220d\") "
Sep 09 20:20:21.566182 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.566106    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9f9e93-d81c-47b2-a0da-a0945065220d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df9f9e93-d81c-47b2-a0da-a0945065220d" (UID: "df9f9e93-d81c-47b2-a0da-a0945065220d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:21.566227 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.566194    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df9f9e93-d81c-47b2-a0da-a0945065220d" (UID: "df9f9e93-d81c-47b2-a0da-a0945065220d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:21.566283 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.566257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df9f9e93-d81c-47b2-a0da-a0945065220d" (UID: "df9f9e93-d81c-47b2-a0da-a0945065220d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:21.568000 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.567981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9f9e93-d81c-47b2-a0da-a0945065220d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df9f9e93-d81c-47b2-a0da-a0945065220d" (UID: "df9f9e93-d81c-47b2-a0da-a0945065220d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:20:21.568102 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.568042    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-api-access-86kwn" (OuterVolumeSpecName: "kube-api-access-86kwn") pod "df9f9e93-d81c-47b2-a0da-a0945065220d" (UID: "df9f9e93-d81c-47b2-a0da-a0945065220d"). InnerVolumeSpecName "kube-api-access-86kwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:21.621383 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.621322    2566 generic.go:356] "Generic (PLEG): container finished" podID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerID="a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc" exitCode=0
Sep 09 20:20:21.621548 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.621397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerDied","Data":"a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc"}
Sep 09 20:20:21.621548 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.621461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg" event={"ID":"df9f9e93-d81c-47b2-a0da-a0945065220d","Type":"ContainerDied","Data":"8d407d64aabd939046e671102ee1719ad7479f9fb0a5d484b840e3ae8575474c"}
Sep 09 20:20:21.621548 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.621414    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"
Sep 09 20:20:21.621548 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.621495    2566 scope.go:117] "RemoveContainer" containerID="bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c"
Sep 09 20:20:21.630524 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.630497    2566 scope.go:117] "RemoveContainer" containerID="a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc"
Sep 09 20:20:21.638760 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.638740    2566 scope.go:117] "RemoveContainer" containerID="3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49"
Sep 09 20:20:21.643879 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.643843    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"]
Sep 09 20:20:21.647335 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.647308    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-drhbg"]
Sep 09 20:20:21.648237 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.648221    2566 scope.go:117] "RemoveContainer" containerID="bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c"
Sep 09 20:20:21.648640 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:20:21.648621    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c\": container with ID starting with bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c not found: ID does not exist" containerID="bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c"
Sep 09 20:20:21.648690 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.648651    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c"} err="failed to get container status \"bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c\": rpc error: code = NotFound desc = could not find container \"bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c\": container with ID starting with bd7b438b3879af9320db838d76cad18912df580fcc330b7842cc5a951174aa2c not found: ID does not exist"
Sep 09 20:20:21.648690 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.648671    2566 scope.go:117] "RemoveContainer" containerID="a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc"
Sep 09 20:20:21.648985 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:20:21.648968    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc\": container with ID starting with a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc not found: ID does not exist" containerID="a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc"
Sep 09 20:20:21.649028 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.648992    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc"} err="failed to get container status \"a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc\": rpc error: code = NotFound desc = could not find container \"a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc\": container with ID starting with a20fe1bea315ef9d0a76c35a65717135ca4a52826341ee42ae9549233087ebbc not found: ID does not exist"
Sep 09 20:20:21.649028 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.649008    2566 scope.go:117] "RemoveContainer" containerID="3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49"
Sep 09 20:20:21.649258 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:20:21.649232    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49\": container with ID starting with 3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49 not found: ID does not exist" containerID="3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49"
Sep 09 20:20:21.649299 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.649261    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49"} err="failed to get container status \"3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49\": rpc error: code = NotFound desc = could not find container \"3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49\": container with ID starting with 3ef95411b8213dd020f68e6b99d2a434cd9d6bcba1264d5955d2d75e04683f49 not found: ID does not exist"
Sep 09 20:20:21.666767 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.666737    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:20:21.666767 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.666769    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86kwn\" (UniqueName: \"kubernetes.io/projected/df9f9e93-d81c-47b2-a0da-a0945065220d-kube-api-access-86kwn\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:20:21.666906 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.666779    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9f9e93-d81c-47b2-a0da-a0945065220d-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:20:21.666906 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.666787    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9f9e93-d81c-47b2-a0da-a0945065220d-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:20:21.666906 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:21.666796    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9f9e93-d81c-47b2-a0da-a0945065220d-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:20:22.471558 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:22.471523    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" path="/var/lib/kubelet/pods/df9f9e93-d81c-47b2-a0da-a0945065220d/volumes"
Sep 09 20:20:23.559694 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.559651    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:23.653106 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653070    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"]
Sep 09 20:20:23.653448 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653435    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="storage-initializer"
Sep 09 20:20:23.653448 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653449    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="storage-initializer"
Sep 09 20:20:23.653554 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653465    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container"
Sep 09 20:20:23.653554 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653472    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container"
Sep 09 20:20:23.653554 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653491    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kube-rbac-proxy"
Sep 09 20:20:23.653554 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653496    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kube-rbac-proxy"
Sep 09 20:20:23.653554 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653546    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kserve-container"
Sep 09 20:20:23.653740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.653563    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9f9e93-d81c-47b2-a0da-a0945065220d" containerName="kube-rbac-proxy"
Sep 09 20:20:23.658148 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.658131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.661153 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.661126    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:20:23.661294 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.661156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:20:23.661400 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.661323    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:20:23.662524 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.662506    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:20:23.662762 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.662746    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:20:23.662848 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.662804    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:20:23.662916 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.662856    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:20:23.667409 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.667384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"]
Sep 09 20:20:23.786956 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.786893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.787732 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.787702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.787947 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.787927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed48fc60-bee2-4909-8ce9-ac3c516b4235-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.793403 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.788204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649jt\" (UniqueName: \"kubernetes.io/projected/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-api-access-649jt\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.793403 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.788256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.889275 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.889275 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.889275 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.889607 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed48fc60-bee2-4909-8ce9-ac3c516b4235-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.889607 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649jt\" (UniqueName: \"kubernetes.io/projected/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-api-access-649jt\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.889776 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.889972 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.890009 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.889983    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.892028 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.891991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed48fc60-bee2-4909-8ce9-ac3c516b4235-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.899030 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.899005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-649jt\" (UniqueName: \"kubernetes.io/projected/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-api-access-649jt\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-rwj44\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:23.970080 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:23.970035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:24.245899 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:24.245874    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"]
Sep 09 20:20:24.247908 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:20:24.247878    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded48fc60_bee2_4909_8ce9_ac3c516b4235.slice/crio-9234e9f2a2cd9e8b645a6f8c8c6158fcc82eae233e91069ccb178a91f5f5b2ad WatchSource:0}: Error finding container 9234e9f2a2cd9e8b645a6f8c8c6158fcc82eae233e91069ccb178a91f5f5b2ad: Status 404 returned error can't find the container with id 9234e9f2a2cd9e8b645a6f8c8c6158fcc82eae233e91069ccb178a91f5f5b2ad
Sep 09 20:20:24.639763 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:24.639677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerStarted","Data":"dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c"}
Sep 09 20:20:24.639763 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:24.639737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerStarted","Data":"9234e9f2a2cd9e8b645a6f8c8c6158fcc82eae233e91069ccb178a91f5f5b2ad"}
Sep 09 20:20:28.659562 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:28.659525    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerID="dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c" exitCode=0
Sep 09 20:20:28.660044 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:28.659610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerDied","Data":"dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c"}
Sep 09 20:20:29.666218 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:29.666184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerStarted","Data":"34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2"}
Sep 09 20:20:29.666632 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:29.666246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerStarted","Data":"1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059"}
Sep 09 20:20:29.666632 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:29.666437    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:29.688719 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:29.688652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podStartSLOduration=2.9823796099999997 podStartE2EDuration="6.6886347s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70625509s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:29.687151217 +0000 UTC m=+1517.714682204" watchObservedRunningTime="2026-09-09 20:20:29.6886347 +0000 UTC m=+1517.716165696"
Sep 09 20:20:30.670910 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:30.670879    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:30.672540 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:30.672509    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:20:31.675636 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:31.675596    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:20:36.679979 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:36.679948    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:20:36.682398 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:36.680447    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:20:46.681087 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:46.681046    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:20:56.680464 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:56.680425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:20:57.318476 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:20:57.318443    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:06.680826 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:06.680782    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:21:16.680464 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:16.680395    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:21:26.680718 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:26.680674    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:21:36.680598 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:36.680557    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:21:46.681762 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:46.681726    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:21:53.792326 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:53.792293    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"]
Sep 09 20:21:53.792826 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:53.792619    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" containerID="cri-o://1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059" gracePeriod=30
Sep 09 20:21:53.792826 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:53.792688    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kube-rbac-proxy" containerID="cri-o://34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2" gracePeriod=30
Sep 09 20:21:53.813116 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:53.813084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:54.059607 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:54.059514    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerID="34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2" exitCode=2
Sep 09 20:21:54.059607 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:54.059589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerDied","Data":"34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2"}
Sep 09 20:21:56.676481 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:56.676428    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 20:21:56.680704 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:56.680672    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:21:58.848080 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.848052    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:21:58.930562 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.930519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kserve-provision-location\") pod \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") "
Sep 09 20:21:58.930750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.930631    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-rbac-proxy-sar-config\") pod \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") "
Sep 09 20:21:58.930750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.930666    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-cabundle-cert\") pod \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") "
Sep 09 20:21:58.930750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.930702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-649jt\" (UniqueName: \"kubernetes.io/projected/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-api-access-649jt\") pod \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") "
Sep 09 20:21:58.930750 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.930732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed48fc60-bee2-4909-8ce9-ac3c516b4235-proxy-tls\") pod \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\" (UID: \"ed48fc60-bee2-4909-8ce9-ac3c516b4235\") "
Sep 09 20:21:58.930967 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.930915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed48fc60-bee2-4909-8ce9-ac3c516b4235" (UID: "ed48fc60-bee2-4909-8ce9-ac3c516b4235"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:21:58.931055 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.931019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed48fc60-bee2-4909-8ce9-ac3c516b4235" (UID: "ed48fc60-bee2-4909-8ce9-ac3c516b4235"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:58.931055 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.931035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed48fc60-bee2-4909-8ce9-ac3c516b4235" (UID: "ed48fc60-bee2-4909-8ce9-ac3c516b4235"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:58.931146 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.931046    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:21:58.933043 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.933019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-api-access-649jt" (OuterVolumeSpecName: "kube-api-access-649jt") pod "ed48fc60-bee2-4909-8ce9-ac3c516b4235" (UID: "ed48fc60-bee2-4909-8ce9-ac3c516b4235"). InnerVolumeSpecName "kube-api-access-649jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:58.933131 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:58.933049    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed48fc60-bee2-4909-8ce9-ac3c516b4235-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed48fc60-bee2-4909-8ce9-ac3c516b4235" (UID: "ed48fc60-bee2-4909-8ce9-ac3c516b4235"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:59.032351 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.032304    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.032351 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.032347    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed48fc60-bee2-4909-8ce9-ac3c516b4235-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.032599 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.032397    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-649jt\" (UniqueName: \"kubernetes.io/projected/ed48fc60-bee2-4909-8ce9-ac3c516b4235-kube-api-access-649jt\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.032599 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.032411    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed48fc60-bee2-4909-8ce9-ac3c516b4235-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:21:59.086118 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.086078    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerID="1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059" exitCode=0
Sep 09 20:21:59.086310 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.086144    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"
Sep 09 20:21:59.086310 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.086152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerDied","Data":"1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059"}
Sep 09 20:21:59.086310 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.086208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44" event={"ID":"ed48fc60-bee2-4909-8ce9-ac3c516b4235","Type":"ContainerDied","Data":"9234e9f2a2cd9e8b645a6f8c8c6158fcc82eae233e91069ccb178a91f5f5b2ad"}
Sep 09 20:21:59.086310 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.086236    2566 scope.go:117] "RemoveContainer" containerID="34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2"
Sep 09 20:21:59.095450 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.095428    2566 scope.go:117] "RemoveContainer" containerID="1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059"
Sep 09 20:21:59.103413 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.103395    2566 scope.go:117] "RemoveContainer" containerID="dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c"
Sep 09 20:21:59.110211 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.110180    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"]
Sep 09 20:21:59.113548 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.113518    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-rwj44"]
Sep 09 20:21:59.114502 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.114482    2566 scope.go:117] "RemoveContainer" containerID="34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2"
Sep 09 20:21:59.114860 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:21:59.114832    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2\": container with ID starting with 34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2 not found: ID does not exist" containerID="34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2"
Sep 09 20:21:59.114965 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.114870    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2"} err="failed to get container status \"34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2\": rpc error: code = NotFound desc = could not find container \"34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2\": container with ID starting with 34842e8f56859b991f3d53b52093144530ac2596f04400ce0b9b5ba1afeb19a2 not found: ID does not exist"
Sep 09 20:21:59.114965 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.114890    2566 scope.go:117] "RemoveContainer" containerID="1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059"
Sep 09 20:21:59.115205 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:21:59.115186    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059\": container with ID starting with 1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059 not found: ID does not exist" containerID="1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059"
Sep 09 20:21:59.115272 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.115216    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059"} err="failed to get container status \"1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059\": rpc error: code = NotFound desc = could not find container \"1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059\": container with ID starting with 1eef6037c677c734ae2212de0773e2704e0121f42d45f681c5cbe5be73296059 not found: ID does not exist"
Sep 09 20:21:59.115272 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.115239    2566 scope.go:117] "RemoveContainer" containerID="dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c"
Sep 09 20:21:59.115553 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:21:59.115536    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c\": container with ID starting with dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c not found: ID does not exist" containerID="dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c"
Sep 09 20:21:59.115605 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:21:59.115562    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c"} err="failed to get container status \"dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c\": rpc error: code = NotFound desc = could not find container \"dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c\": container with ID starting with dfe0ae6051658e10c7626cd7820f003990d4777aa42b5ae75a20ce9c9af7946c not found: ID does not exist"
Sep 09 20:22:00.470429 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:00.470383    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" path="/var/lib/kubelet/pods/ed48fc60-bee2-4909-8ce9-ac3c516b4235/volumes"
Sep 09 20:22:07.999681 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:07.999589    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:08.046851 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.046780    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"]
Sep 09 20:22:08.047253 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047238    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kube-rbac-proxy"
Sep 09 20:22:08.047324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047255    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kube-rbac-proxy"
Sep 09 20:22:08.047324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047267    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container"
Sep 09 20:22:08.047324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047272    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container"
Sep 09 20:22:08.047324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047282    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="storage-initializer"
Sep 09 20:22:08.047324 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047291    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="storage-initializer"
Sep 09 20:22:08.047610 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047350    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kserve-container"
Sep 09 20:22:08.047610 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.047378    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed48fc60-bee2-4909-8ce9-ac3c516b4235" containerName="kube-rbac-proxy"
Sep 09 20:22:08.051949 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.051934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.054577 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.054552    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 09 20:22:08.054870 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.054854    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 09 20:22:08.055973 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.055952    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:22:08.056068 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.055969    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:22:08.056068 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.055954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:22:08.060168 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.060142    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"]
Sep 09 20:22:08.179540 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.179513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:08.215975 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.215944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b214faa-64f2-48b2-b058-33a9dd1f978d-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.216096 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.215991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qbl\" (UniqueName: \"kubernetes.io/projected/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-api-access-89qbl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.216096 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.216054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.317337 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.317251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b214faa-64f2-48b2-b058-33a9dd1f978d-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.317337 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.317300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qbl\" (UniqueName: \"kubernetes.io/projected/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-api-access-89qbl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.317593 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.317344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.318068 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.317997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.319871 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.319849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b214faa-64f2-48b2-b058-33a9dd1f978d-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.326581 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.326559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qbl\" (UniqueName: \"kubernetes.io/projected/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-api-access-89qbl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xjj5k\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.363375 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.363336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:08.638381 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.638341    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"]
Sep 09 20:22:08.640928 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:22:08.640899    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b214faa_64f2_48b2_b058_33a9dd1f978d.slice/crio-b31e5433698ada7a97d757ef48d862027ea5d99ebb200d3ee4ed0d0b06ae660b WatchSource:0}: Error finding container b31e5433698ada7a97d757ef48d862027ea5d99ebb200d3ee4ed0d0b06ae660b: Status 404 returned error can't find the container with id b31e5433698ada7a97d757ef48d862027ea5d99ebb200d3ee4ed0d0b06ae660b
Sep 09 20:22:08.642961 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:08.642943    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:22:09.134265 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:09.134232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" event={"ID":"8b214faa-64f2-48b2-b058-33a9dd1f978d","Type":"ContainerStarted","Data":"b31e5433698ada7a97d757ef48d862027ea5d99ebb200d3ee4ed0d0b06ae660b"}
Sep 09 20:22:10.140811 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:10.140727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" event={"ID":"8b214faa-64f2-48b2-b058-33a9dd1f978d","Type":"ContainerStarted","Data":"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"}
Sep 09 20:22:10.140811 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:10.140797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" event={"ID":"8b214faa-64f2-48b2-b058-33a9dd1f978d","Type":"ContainerStarted","Data":"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"}
Sep 09 20:22:10.141233 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:10.140880    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:10.164669 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:10.164619    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" podStartSLOduration=0.994928818 podStartE2EDuration="2.164598801s" totalImagesPullingTime="1.169669983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:10.162452805 +0000 UTC m=+1618.189983792" watchObservedRunningTime="2026-09-09 20:22:10.164598801 +0000 UTC m=+1618.192129786"
Sep 09 20:22:11.145821 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:11.145781    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:11.147731 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:11.147709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:18.160864 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:18.160832    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:22:28.050290 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.050254    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:28.085673 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.085631    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"]
Sep 09 20:22:28.093547 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.093518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.096954 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.096918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:22:28.096954 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.096947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:22:28.097129 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.096923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 09 20:22:28.097129 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.096918    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 09 20:22:28.103124 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.103089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"]
Sep 09 20:22:28.176787 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.176749    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:28.189995 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.189963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.190090 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.190035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.190090 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.190060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.190090 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.190083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d254\" (UniqueName: \"kubernetes.io/projected/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-api-access-4d254\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.190193 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.190144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b81f302-3f54-44c9-a582-e1f8e534bb5f-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.291043 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.291214 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.291214 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.291214 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d254\" (UniqueName: \"kubernetes.io/projected/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-api-access-4d254\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.291214 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b81f302-3f54-44c9-a582-e1f8e534bb5f-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.291538 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.291964 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.292015 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.291969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.293781 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.293763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b81f302-3f54-44c9-a582-e1f8e534bb5f-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.301435 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.301379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d254\" (UniqueName: \"kubernetes.io/projected/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-api-access-4d254\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-7fxrk\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.404971 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.404911    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:28.684542 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:28.684451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"]
Sep 09 20:22:28.687461 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:22:28.687428    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b81f302_3f54_44c9_a582_e1f8e534bb5f.slice/crio-c95a9e2e621c18915d7e5e70e6151894c6fa855ca2a2dafd5fdad24648602f80 WatchSource:0}: Error finding container c95a9e2e621c18915d7e5e70e6151894c6fa855ca2a2dafd5fdad24648602f80: Status 404 returned error can't find the container with id c95a9e2e621c18915d7e5e70e6151894c6fa855ca2a2dafd5fdad24648602f80
Sep 09 20:22:29.239663 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:29.239621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerStarted","Data":"5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8"}
Sep 09 20:22:29.240036 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:29.239730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerStarted","Data":"c95a9e2e621c18915d7e5e70e6151894c6fa855ca2a2dafd5fdad24648602f80"}
Sep 09 20:22:33.262202 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:33.262161    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerID="5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8" exitCode=0
Sep 09 20:22:33.262647 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:33.262234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerDied","Data":"5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8"}
Sep 09 20:22:34.269802 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:34.269771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerStarted","Data":"4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2"}
Sep 09 20:22:34.270230 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:34.269837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerStarted","Data":"6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b"}
Sep 09 20:22:34.270230 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:34.269860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerStarted","Data":"2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0"}
Sep 09 20:22:34.270230 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:34.270147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:34.270414 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:34.270274    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:34.271752 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:34.271725    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:22:34.305850 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:34.305792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podStartSLOduration=2.389937418 podStartE2EDuration="6.305772991s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.915835573s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:34.303429621 +0000 UTC m=+1642.330960607" watchObservedRunningTime="2026-09-09 20:22:34.305772991 +0000 UTC m=+1642.333303977"
Sep 09 20:22:35.274935 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:35.274900    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:35.275427 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:35.275087    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:22:35.276285 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:35.276255    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:36.280122 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:36.280079    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:22:36.280707 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:36.280683    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:41.287034 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:41.287006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:22:41.287658 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:41.287626    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:22:41.288047 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:41.288012    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:51.287997 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:51.287957    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:22:51.288590 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:22:51.288563    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:01.287706 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:01.287659    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:23:01.288151 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:01.288125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:10.336590 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:10.336552    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:11.287759 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:11.287716    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:23:11.288355 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:11.288332    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:21.288157 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:21.288106    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:23:21.288692 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:21.288513    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:31.288632 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:31.288580    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:23:31.289091 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:31.289006    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:41.288221 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:41.288125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:23:41.288747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:41.288719    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:51.288572 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:51.288538    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:23:51.289063 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:23:51.288634    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:24:03.180073 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.180041    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-xjj5k_8b214faa-64f2-48b2-b058-33a9dd1f978d/kserve-container/0.log"
Sep 09 20:24:03.374892 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.374860    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"]
Sep 09 20:24:03.375203 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.375170    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" containerID="cri-o://2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0" gracePeriod=30
Sep 09 20:24:03.375348 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.375218    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy" containerID="cri-o://6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b" gracePeriod=30
Sep 09 20:24:03.375348 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.375170    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" containerID="cri-o://4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2" gracePeriod=30
Sep 09 20:24:03.398617 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.398585    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:03.403755 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.403730    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"]
Sep 09 20:24:03.404007 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.403983    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kserve-container" containerID="cri-o://cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c" gracePeriod=30
Sep 09 20:24:03.404099 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.403995    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kube-rbac-proxy" containerID="cri-o://992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b" gracePeriod=30
Sep 09 20:24:03.495308 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.495275    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:03.648933 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.648901    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:24:03.693678 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.693641    2566 generic.go:356] "Generic (PLEG): container finished" podID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerID="992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b" exitCode=2
Sep 09 20:24:03.693678 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.693668    2566 generic.go:356] "Generic (PLEG): container finished" podID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerID="cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c" exitCode=2
Sep 09 20:24:03.693920 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.693689    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"
Sep 09 20:24:03.693920 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.693718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" event={"ID":"8b214faa-64f2-48b2-b058-33a9dd1f978d","Type":"ContainerDied","Data":"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"}
Sep 09 20:24:03.693920 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.693776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" event={"ID":"8b214faa-64f2-48b2-b058-33a9dd1f978d","Type":"ContainerDied","Data":"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"}
Sep 09 20:24:03.693920 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.693799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k" event={"ID":"8b214faa-64f2-48b2-b058-33a9dd1f978d","Type":"ContainerDied","Data":"b31e5433698ada7a97d757ef48d862027ea5d99ebb200d3ee4ed0d0b06ae660b"}
Sep 09 20:24:03.693920 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.693826    2566 scope.go:117] "RemoveContainer" containerID="992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"
Sep 09 20:24:03.696547 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.696524    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerID="6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b" exitCode=2
Sep 09 20:24:03.696682 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.696599    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerDied","Data":"6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b"}
Sep 09 20:24:03.702797 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.702779    2566 scope.go:117] "RemoveContainer" containerID="cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"
Sep 09 20:24:03.710635 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.710616    2566 scope.go:117] "RemoveContainer" containerID="992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"
Sep 09 20:24:03.710925 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:24:03.710909    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b\": container with ID starting with 992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b not found: ID does not exist" containerID="992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"
Sep 09 20:24:03.710993 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.710934    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"} err="failed to get container status \"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b\": rpc error: code = NotFound desc = could not find container \"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b\": container with ID starting with 992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b not found: ID does not exist"
Sep 09 20:24:03.710993 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.710951    2566 scope.go:117] "RemoveContainer" containerID="cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"
Sep 09 20:24:03.711257 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:24:03.711233    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c\": container with ID starting with cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c not found: ID does not exist" containerID="cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"
Sep 09 20:24:03.711298 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.711266    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"} err="failed to get container status \"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c\": rpc error: code = NotFound desc = could not find container \"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c\": container with ID starting with cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c not found: ID does not exist"
Sep 09 20:24:03.711298 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.711283    2566 scope.go:117] "RemoveContainer" containerID="992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"
Sep 09 20:24:03.711573 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.711549    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b"} err="failed to get container status \"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b\": rpc error: code = NotFound desc = could not find container \"992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b\": container with ID starting with 992a6f39e1ca4658a5ebd5a69d08703f1e705a03fd7b8cf8d1bedf170ba1157b not found: ID does not exist"
Sep 09 20:24:03.711642 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.711574    2566 scope.go:117] "RemoveContainer" containerID="cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"
Sep 09 20:24:03.711836 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.711818    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c"} err="failed to get container status \"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c\": rpc error: code = NotFound desc = could not find container \"cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c\": container with ID starting with cd832bb6e6d8d5e2e97c4f82e47e7bfff6a87c77f6a105475a0fb780a11c2b1c not found: ID does not exist"
Sep 09 20:24:03.739182 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.739157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89qbl\" (UniqueName: \"kubernetes.io/projected/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-api-access-89qbl\") pod \"8b214faa-64f2-48b2-b058-33a9dd1f978d\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") "
Sep 09 20:24:03.739289 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.739210    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b214faa-64f2-48b2-b058-33a9dd1f978d-proxy-tls\") pod \"8b214faa-64f2-48b2-b058-33a9dd1f978d\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") "
Sep 09 20:24:03.739289 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.739230    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-rbac-proxy-sar-config\") pod \"8b214faa-64f2-48b2-b058-33a9dd1f978d\" (UID: \"8b214faa-64f2-48b2-b058-33a9dd1f978d\") "
Sep 09 20:24:03.739643 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.739617    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b214faa-64f2-48b2-b058-33a9dd1f978d" (UID: "8b214faa-64f2-48b2-b058-33a9dd1f978d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:03.741616 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.741589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-api-access-89qbl" (OuterVolumeSpecName: "kube-api-access-89qbl") pod "8b214faa-64f2-48b2-b058-33a9dd1f978d" (UID: "8b214faa-64f2-48b2-b058-33a9dd1f978d"). InnerVolumeSpecName "kube-api-access-89qbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:03.741701 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.741682    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b214faa-64f2-48b2-b058-33a9dd1f978d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b214faa-64f2-48b2-b058-33a9dd1f978d" (UID: "8b214faa-64f2-48b2-b058-33a9dd1f978d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:24:03.840677 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.840593    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qbl\" (UniqueName: \"kubernetes.io/projected/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-api-access-89qbl\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:03.840677 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.840626    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b214faa-64f2-48b2-b058-33a9dd1f978d-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:03.840677 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:03.840636    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b214faa-64f2-48b2-b058-33a9dd1f978d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:04.016350 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:04.016316    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"]
Sep 09 20:24:04.019894 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:04.019867    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xjj5k"]
Sep 09 20:24:04.330012 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:04.329968    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:04.431342 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:04.431307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:04.471202 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:04.471163    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" path="/var/lib/kubelet/pods/8b214faa-64f2-48b2-b058-33a9dd1f978d/volumes"
Sep 09 20:24:06.280694 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:06.280658    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:24:08.724961 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:08.724914    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerID="2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0" exitCode=0
Sep 09 20:24:08.725476 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:08.724988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerDied","Data":"2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0"}
Sep 09 20:24:11.281380 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:11.281308    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:24:11.287805 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:11.287770    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:24:11.288158 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:11.288128    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:16.280896 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:16.280855    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:24:16.281333 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:16.281008    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:24:20.231855 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:20.231829    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:20.257571 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:20.257532    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:20.268254 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:20.268230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:20.279034 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:20.279003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:21.124820 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.124792    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:21.137506 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.137478    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:21.147773 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.147744    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:21.157566 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.157542    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:21.281010 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.280972    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:24:21.288215 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.288183    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:24:21.288569 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.288549    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:21.991746 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:21.991716    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:22.004109 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:22.004084    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:22.021177 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:22.021154    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:22.032156 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:22.032134    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:22.888103 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:22.888070    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:22.900353 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:22.900327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:22.910129 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:22.910105    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:22.920715 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:22.920690    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:23.731236 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:23.731204    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:23.743994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:23.743971    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:23.756035 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:23.756012    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:23.766140 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:23.766110    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:24.596326 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:24.596301    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:24.610201 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:24.610178    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:24.622178 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:24.622155    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:24.633963 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:24.633935    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:25.459993 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:25.459961    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:25.471101 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:25.471075    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:25.482978 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:25.482957    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:25.492905 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:25.492883    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:26.280508 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:26.280470    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:24:26.312245 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:26.312222    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:26.327744 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:26.327721    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:26.339270 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:26.339240    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:26.350793 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:26.350766    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:27.167425 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:27.167397    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:27.181351 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:27.181320    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:27.192640 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:27.192611    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:27.204480 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:27.204454    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:28.047570 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.047543    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:28.059097 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.059075    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:28.070354 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.070329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:28.080353 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.080335    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:28.909711 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.909688    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:28.922543 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.922517    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:28.932600 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.932575    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:28.946595 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:28.946574    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:29.771853 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:29.771825    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:29.783624 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:29.783595    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:29.794001 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:29.793980    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:29.804629 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:29.804609    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:30.672051 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:30.672024    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:30.683812 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:30.683790    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:30.693382 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:30.693340    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:30.703319 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:30.703293    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:31.280843 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.280800    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:24:31.287715 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.287676    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:24:31.287864 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.287826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:24:31.288030 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.288005    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:31.288139 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.288125    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:24:31.561943 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.561863    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kserve-container/0.log"
Sep 09 20:24:31.574131 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.574107    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/kube-rbac-proxy/0.log"
Sep 09 20:24:31.584192 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.584163    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/agent/0.log"
Sep 09 20:24:31.593979 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:31.593953    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-7fxrk_9b81f302-3f54-44c9-a582-e1f8e534bb5f/storage-initializer/0.log"
Sep 09 20:24:33.527492 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.527468    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:24:33.606064 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606023    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b81f302-3f54-44c9-a582-e1f8e534bb5f-proxy-tls\") pod \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") "
Sep 09 20:24:33.606265 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606087    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kserve-provision-location\") pod \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") "
Sep 09 20:24:33.606265 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606124    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-cabundle-cert\") pod \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") "
Sep 09 20:24:33.606265 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606184    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d254\" (UniqueName: \"kubernetes.io/projected/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-api-access-4d254\") pod \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") "
Sep 09 20:24:33.606265 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606207    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-rbac-proxy-sar-config\") pod \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\" (UID: \"9b81f302-3f54-44c9-a582-e1f8e534bb5f\") "
Sep 09 20:24:33.606534 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b81f302-3f54-44c9-a582-e1f8e534bb5f" (UID: "9b81f302-3f54-44c9-a582-e1f8e534bb5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:33.606641 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606609    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b81f302-3f54-44c9-a582-e1f8e534bb5f" (UID: "9b81f302-3f54-44c9-a582-e1f8e534bb5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:33.606641 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.606619    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b81f302-3f54-44c9-a582-e1f8e534bb5f" (UID: "9b81f302-3f54-44c9-a582-e1f8e534bb5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:33.608301 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.608282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b81f302-3f54-44c9-a582-e1f8e534bb5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b81f302-3f54-44c9-a582-e1f8e534bb5f" (UID: "9b81f302-3f54-44c9-a582-e1f8e534bb5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:24:33.608416 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.608380    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-api-access-4d254" (OuterVolumeSpecName: "kube-api-access-4d254") pod "9b81f302-3f54-44c9-a582-e1f8e534bb5f" (UID: "9b81f302-3f54-44c9-a582-e1f8e534bb5f"). InnerVolumeSpecName "kube-api-access-4d254". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:33.707781 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.707688    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d254\" (UniqueName: \"kubernetes.io/projected/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-api-access-4d254\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:33.707781 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.707728    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:33.707781 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.707745    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b81f302-3f54-44c9-a582-e1f8e534bb5f-proxy-tls\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:33.707781 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.707759    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b81f302-3f54-44c9-a582-e1f8e534bb5f-kserve-provision-location\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:33.707781 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.707772    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b81f302-3f54-44c9-a582-e1f8e534bb5f-cabundle-cert\") on node \"ip-10-0-135-134.ec2.internal\" DevicePath \"\""
Sep 09 20:24:33.846827 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.846786    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerID="4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2" exitCode=0
Sep 09 20:24:33.847002 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.846873    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"
Sep 09 20:24:33.847002 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.846867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerDied","Data":"4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2"}
Sep 09 20:24:33.847097 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.847034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk" event={"ID":"9b81f302-3f54-44c9-a582-e1f8e534bb5f","Type":"ContainerDied","Data":"c95a9e2e621c18915d7e5e70e6151894c6fa855ca2a2dafd5fdad24648602f80"}
Sep 09 20:24:33.847097 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.847070    2566 scope.go:117] "RemoveContainer" containerID="4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2"
Sep 09 20:24:33.856346 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.856324    2566 scope.go:117] "RemoveContainer" containerID="6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b"
Sep 09 20:24:33.864697 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.864682    2566 scope.go:117] "RemoveContainer" containerID="2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0"
Sep 09 20:24:33.869332 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.869295    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"]
Sep 09 20:24:33.874157 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.874135    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-7fxrk"]
Sep 09 20:24:33.874230 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.874144    2566 scope.go:117] "RemoveContainer" containerID="5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8"
Sep 09 20:24:33.882090 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.882070    2566 scope.go:117] "RemoveContainer" containerID="4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2"
Sep 09 20:24:33.882447 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:24:33.882426    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2\": container with ID starting with 4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2 not found: ID does not exist" containerID="4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2"
Sep 09 20:24:33.882544 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.882458    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2"} err="failed to get container status \"4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2\": rpc error: code = NotFound desc = could not find container \"4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2\": container with ID starting with 4c69c29b08689d27c4f76fca6f6d3010cf70fb03a9dd02659b1e09486b5a38c2 not found: ID does not exist"
Sep 09 20:24:33.882544 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.882483    2566 scope.go:117] "RemoveContainer" containerID="6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b"
Sep 09 20:24:33.882799 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:24:33.882782    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b\": container with ID starting with 6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b not found: ID does not exist" containerID="6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b"
Sep 09 20:24:33.882850 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.882805    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b"} err="failed to get container status \"6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b\": rpc error: code = NotFound desc = could not find container \"6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b\": container with ID starting with 6d845304cc904d0447c143193ead8f59b9ddde3f4acae402b2107c69b6a7731b not found: ID does not exist"
Sep 09 20:24:33.882850 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.882822    2566 scope.go:117] "RemoveContainer" containerID="2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0"
Sep 09 20:24:33.883079 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:24:33.883061    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0\": container with ID starting with 2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0 not found: ID does not exist" containerID="2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0"
Sep 09 20:24:33.883136 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.883084    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0"} err="failed to get container status \"2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0\": rpc error: code = NotFound desc = could not find container \"2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0\": container with ID starting with 2d9a5e6cde08da2ef34f867fa2e5353d9f8a9e7289e9522c6eefe2fe190cf6d0 not found: ID does not exist"
Sep 09 20:24:33.883136 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.883101    2566 scope.go:117] "RemoveContainer" containerID="5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8"
Sep 09 20:24:33.883338 ip-10-0-135-134 kubenswrapper[2566]: E0909 20:24:33.883319    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8\": container with ID starting with 5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8 not found: ID does not exist" containerID="5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8"
Sep 09 20:24:33.883436 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:33.883347    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8"} err="failed to get container status \"5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8\": rpc error: code = NotFound desc = could not find container \"5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8\": container with ID starting with 5826404079fc31524e827296bf0e317716c04c109efa27f2ed904b0853a920c8 not found: ID does not exist"
Sep 09 20:24:34.470280 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:34.470244    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" path="/var/lib/kubelet/pods/9b81f302-3f54-44c9-a582-e1f8e534bb5f/volumes"
Sep 09 20:24:35.478852 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:35.478823    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:35.655343 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:35.655312    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:35.758499 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:35.758415    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:38.684952 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:38.684922    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nfgkh_a45ab428-78e1-4ecc-8c0e-4bdddc2ddb36/global-pull-secret-syncer/0.log"
Sep 09 20:24:38.719142 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:38.719114    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log"
Sep 09 20:24:38.813757 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:38.813724    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-134.ec2.internal_bc4c3568a288696fcc9b2d4eeb0ba150/haproxy/0.log"
Sep 09 20:24:41.250400 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:41.250350    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:41.306214 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:41.306183    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jc5l_a663e806-1533-458e-bc6c-87e25425a20e/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:43.187259 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.187230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4945bfb-c5d8-4b61-93e1-e4f69e7879d6/alertmanager/0.log"
Sep 09 20:24:43.217270 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.217245    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4945bfb-c5d8-4b61-93e1-e4f69e7879d6/config-reloader/0.log"
Sep 09 20:24:43.245948 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.245920    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4945bfb-c5d8-4b61-93e1-e4f69e7879d6/kube-rbac-proxy-web/0.log"
Sep 09 20:24:43.275938 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.275911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4945bfb-c5d8-4b61-93e1-e4f69e7879d6/kube-rbac-proxy/0.log"
Sep 09 20:24:43.311799 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.311769    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4945bfb-c5d8-4b61-93e1-e4f69e7879d6/kube-rbac-proxy-metric/0.log"
Sep 09 20:24:43.340322 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.340290    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4945bfb-c5d8-4b61-93e1-e4f69e7879d6/prom-label-proxy/0.log"
Sep 09 20:24:43.366107 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.366081    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c4945bfb-c5d8-4b61-93e1-e4f69e7879d6/init-config-reloader/0.log"
Sep 09 20:24:43.442154 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.442056    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-vxnm9_b2884be2-bbc7-4511-a78d-9933573cee86/kube-state-metrics/0.log"
Sep 09 20:24:43.464956 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.464928    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-vxnm9_b2884be2-bbc7-4511-a78d-9933573cee86/kube-rbac-proxy-main/0.log"
Sep 09 20:24:43.489735 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.489706    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-vxnm9_b2884be2-bbc7-4511-a78d-9933573cee86/kube-rbac-proxy-self/0.log"
Sep 09 20:24:43.520244 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.520220    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5475f4dd97-ldk2k_ed369a29-24d0-4290-ad74-c7b9cab47ba1/metrics-server/0.log"
Sep 09 20:24:43.588754 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.588728    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5mbwq_99e876b6-dc17-4c45-bf5d-2ee473f6bf73/node-exporter/0.log"
Sep 09 20:24:43.617402 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.617375    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5mbwq_99e876b6-dc17-4c45-bf5d-2ee473f6bf73/kube-rbac-proxy/0.log"
Sep 09 20:24:43.641567 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.641539    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5mbwq_99e876b6-dc17-4c45-bf5d-2ee473f6bf73/init-textfile/0.log"
Sep 09 20:24:43.864843 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.864813    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-d26g4_8f8f3e40-bf9e-415d-86b9-9142516823e3/kube-rbac-proxy-main/0.log"
Sep 09 20:24:43.890505 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.890479    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-d26g4_8f8f3e40-bf9e-415d-86b9-9142516823e3/kube-rbac-proxy-self/0.log"
Sep 09 20:24:43.916703 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:43.916677    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-d26g4_8f8f3e40-bf9e-415d-86b9-9142516823e3/openshift-state-metrics/0.log"
Sep 09 20:24:44.199341 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:44.199264    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-j9gk7_ba7cb7b9-a4f9-416a-8ead-62cf7c089b8d/prometheus-operator-admission-webhook/0.log"
Sep 09 20:24:44.229833 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:44.229805    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d5f9b79f-4qrw4_7eb5bb4f-55bc-4d64-9348-0e82040b8fda/telemeter-client/0.log"
Sep 09 20:24:44.260833 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:44.260805    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d5f9b79f-4qrw4_7eb5bb4f-55bc-4d64-9348-0e82040b8fda/reload/0.log"
Sep 09 20:24:44.287843 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:44.287808    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d5f9b79f-4qrw4_7eb5bb4f-55bc-4d64-9348-0e82040b8fda/kube-rbac-proxy/0.log"
Sep 09 20:24:46.252490 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.252464    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/1.log"
Sep 09 20:24:46.258284 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.258260    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2ztwg_d06edb24-e8c4-4b7b-8c87-0f3181a3a1c3/console-operator/2.log"
Sep 09 20:24:46.682706 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.682631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c7d65d5c7-kdl2n_2532c5d5-9d93-44bf-8d3c-c21e54bcca38/console/0.log"
Sep 09 20:24:46.715430 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.715406    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-ctbm8_85dabe45-811b-4f26-80e0-e6c00f8a8048/download-server/0.log"
Sep 09 20:24:46.820322 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820287    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"]
Sep 09 20:24:46.820747 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820731    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820749    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820759    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kube-rbac-proxy"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820765    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kube-rbac-proxy"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820777    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="storage-initializer"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="storage-initializer"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820788    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820793    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820802    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820807    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820818    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kserve-container"
Sep 09 20:24:46.820800 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820823    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kserve-container"
Sep 09 20:24:46.821292 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820873    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kube-rbac-proxy"
Sep 09 20:24:46.821292 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820882    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kserve-container"
Sep 09 20:24:46.821292 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820888    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="kserve-container"
Sep 09 20:24:46.821292 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820893    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b81f302-3f54-44c9-a582-e1f8e534bb5f" containerName="agent"
Sep 09 20:24:46.821292 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.820900    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b214faa-64f2-48b2-b058-33a9dd1f978d" containerName="kube-rbac-proxy"
Sep 09 20:24:46.824083 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.824064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:46.826827 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.826803    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwwj7\"/\"openshift-service-ca.crt\""
Sep 09 20:24:46.828200 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.828179    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cwwj7\"/\"default-dockercfg-54whg\""
Sep 09 20:24:46.828315 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.828185    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwwj7\"/\"kube-root-ca.crt\""
Sep 09 20:24:46.832595 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.832570    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"]
Sep 09 20:24:46.920461 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.920433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2sp\" (UniqueName: \"kubernetes.io/projected/f57ce462-a57a-4131-abb3-7f8fa2803b43-kube-api-access-6z2sp\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:46.920644 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.920487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-lib-modules\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:46.920644 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.920554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-proc\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:46.920644 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.920582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-sys\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:46.920644 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:46.920614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-podres\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021780 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-lib-modules\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-proc\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-sys\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-podres\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2sp\" (UniqueName: \"kubernetes.io/projected/f57ce462-a57a-4131-abb3-7f8fa2803b43-kube-api-access-6z2sp\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-proc\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-sys\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-lib-modules\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.021994 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.021990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f57ce462-a57a-4131-abb3-7f8fa2803b43-podres\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.030533 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.030509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2sp\" (UniqueName: \"kubernetes.io/projected/f57ce462-a57a-4131-abb3-7f8fa2803b43-kube-api-access-6z2sp\") pod \"perf-node-gather-daemonset-95w4b\" (UID: \"f57ce462-a57a-4131-abb3-7f8fa2803b43\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.134786 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.134744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.406872 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.406830    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"]
Sep 09 20:24:47.409637 ip-10-0-135-134 kubenswrapper[2566]: W0909 20:24:47.409604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf57ce462_a57a_4131_abb3_7f8fa2803b43.slice/crio-a074a8e632b671fa73b48104da666159a11af235e12085c1d8ae496259775fe4 WatchSource:0}: Error finding container a074a8e632b671fa73b48104da666159a11af235e12085c1d8ae496259775fe4: Status 404 returned error can't find the container with id a074a8e632b671fa73b48104da666159a11af235e12085c1d8ae496259775fe4
Sep 09 20:24:47.917738 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.917697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b" event={"ID":"f57ce462-a57a-4131-abb3-7f8fa2803b43","Type":"ContainerStarted","Data":"612489a6fe17e9d621497b23df9a5ed8716a6515b3892e562c6ab862852d3726"}
Sep 09 20:24:47.917947 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.917776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b" event={"ID":"f57ce462-a57a-4131-abb3-7f8fa2803b43","Type":"ContainerStarted","Data":"a074a8e632b671fa73b48104da666159a11af235e12085c1d8ae496259775fe4"}
Sep 09 20:24:47.917947 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.917847    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:47.941135 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:47.941083    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b" podStartSLOduration=1.9410667849999999 podStartE2EDuration="1.941066785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:47.938903407 +0000 UTC m=+1775.966434392" watchObservedRunningTime="2026-09-09 20:24:47.941066785 +0000 UTC m=+1775.968597770"
Sep 09 20:24:48.402692 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:48.402662    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xlnws_5788617e-c54d-47ca-811f-4b2fb0ec2f68/dns/0.log"
Sep 09 20:24:48.427293 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:48.427256    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xlnws_5788617e-c54d-47ca-811f-4b2fb0ec2f68/kube-rbac-proxy/0.log"
Sep 09 20:24:48.507295 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:48.507259    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zwq9h_e9da6979-551d-44b6-bce5-7ecdc4c97ca6/dns-node-resolver/0.log"
Sep 09 20:24:49.049515 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:49.049486    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rjq67_8e25d865-79c7-497b-af66-104fda1b243d/node-ca/0.log"
Sep 09 20:24:50.219542 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:50.219512    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgv5q_422d7108-e3a0-41bd-9f8c-cdb796cf13ae/serve-healthcheck-canary/0.log"
Sep 09 20:24:50.775026 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:50.774999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnrrp_ed662548-084a-408b-8fa2-44a56d36a7ae/kube-rbac-proxy/0.log"
Sep 09 20:24:50.801717 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:50.801691    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnrrp_ed662548-084a-408b-8fa2-44a56d36a7ae/exporter/0.log"
Sep 09 20:24:50.826570 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:50.826541    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qnrrp_ed662548-084a-408b-8fa2-44a56d36a7ae/extractor/0.log"
Sep 09 20:24:53.178833 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:53.178804    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-9hf86_6d357f1d-90f1-4f3f-99e8-486cf754a8ec/server/0.log"
Sep 09 20:24:53.300566 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:53.300536    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-gxfqb_a5a8996a-eec1-43e4-97e4-a2b337f6012c/manager/0.log"
Sep 09 20:24:53.386164 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:53.386134    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-nmxv2_c5f94662-17ae-4671-9bc3-89c2334927e6/seaweedfs/0.log"
Sep 09 20:24:53.932116 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:53.932085    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-95w4b"
Sep 09 20:24:59.329825 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.329799    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f8tlb_e4d3e68d-ec2f-4040-aacf-0b394ca7aaea/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:24:59.354713 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.354686    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f8tlb_e4d3e68d-ec2f-4040-aacf-0b394ca7aaea/egress-router-binary-copy/0.log"
Sep 09 20:24:59.377035 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.377009    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f8tlb_e4d3e68d-ec2f-4040-aacf-0b394ca7aaea/cni-plugins/0.log"
Sep 09 20:24:59.401194 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.401167    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f8tlb_e4d3e68d-ec2f-4040-aacf-0b394ca7aaea/bond-cni-plugin/0.log"
Sep 09 20:24:59.424254 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.424228    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f8tlb_e4d3e68d-ec2f-4040-aacf-0b394ca7aaea/routeoverride-cni/0.log"
Sep 09 20:24:59.447900 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.447851    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f8tlb_e4d3e68d-ec2f-4040-aacf-0b394ca7aaea/whereabouts-cni-bincopy/0.log"
Sep 09 20:24:59.473002 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.472976    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f8tlb_e4d3e68d-ec2f-4040-aacf-0b394ca7aaea/whereabouts-cni/0.log"
Sep 09 20:24:59.751399 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.751347    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lkr8t_dc94d9fd-1190-4cdd-b62e-a7700bde4772/kube-multus/0.log"
Sep 09 20:24:59.915302 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.915274    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s6f8r_b2014487-2cbf-4ff3-a134-4a70e6b9a002/network-metrics-daemon/0.log"
Sep 09 20:24:59.936754 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:24:59.936712    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s6f8r_b2014487-2cbf-4ff3-a134-4a70e6b9a002/kube-rbac-proxy/0.log"
Sep 09 20:25:00.698517 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.698489    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/ovn-controller/0.log"
Sep 09 20:25:00.719058 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.719027    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/ovn-acl-logging/0.log"
Sep 09 20:25:00.745397 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.745346    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/kube-rbac-proxy-node/0.log"
Sep 09 20:25:00.768926 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.768885    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:25:00.790740 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.790713    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/northd/0.log"
Sep 09 20:25:00.814154 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.814123    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/nbdb/0.log"
Sep 09 20:25:00.837344 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.837319    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/sbdb/0.log"
Sep 09 20:25:00.883587 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:00.883554    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4pfft_d3ddca81-6886-4d4e-b27b-b22c22b1e405/ovnkube-controller/0.log"
Sep 09 20:25:02.384812 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:02.384774    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qk86w_4ddf9d9b-99c2-4e54-8d05-a9df36d22b2b/network-check-target-container/0.log"
Sep 09 20:25:03.394432 ip-10-0-135-134 kubenswrapper[2566]: I0909 20:25:03.394342    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-942zz_e82940f2-81aa-47cb-8e14-dd59d258a2f7/iptables-alerter/0.log"