Jul 21 12:11:13.717278 ip-10-0-135-97 systemd[1]: Starting Kubernetes Kubelet... Jul 21 12:11:14.103578 ip-10-0-135-97 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 12:11:14.103578 ip-10-0-135-97 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 12:11:14.103578 ip-10-0-135-97 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 12:11:14.103578 ip-10-0-135-97 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 12:11:14.103578 ip-10-0-135-97 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 12:11:14.104454 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.103646 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 12:11:14.108778 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108751 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:11:14.108778 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108773 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:11:14.108778 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108779 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:11:14.108778 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108784 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:11:14.108778 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108788 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108794 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108799 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108803 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108807 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108812 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108816 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108821 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108824 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108828 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108832 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108835 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108839 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108844 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108869 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108877 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108882 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108886 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108890 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108895 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:11:14.109138 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108899 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108905 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108910 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108915 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108920 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108924 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108928 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108933 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108937 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108941 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108945 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108949 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108955 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108960 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108966 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108970 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108975 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108980 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108984 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:11:14.110010 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108988 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108993 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.108998 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109003 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109009 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109016 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109041 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109046 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109069 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109075 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109080 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109084 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109089 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109093 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109098 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109103 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109107 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109112 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109116 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:11:14.110876 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109121 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109125 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109130 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109134 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109138 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109142 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109148 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109153 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109159 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109164 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109169 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109173 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109178 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109185 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109192 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109197 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109202 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109207 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109217 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:11:14.111769 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109223 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109241 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109248 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109254 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109258 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109263 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109268 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109272 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109276 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109282 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109287 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109292 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109297 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109302 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109306 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109310 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109315 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.109320 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110654 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110665 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:11:14.112643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110670 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110680 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110687 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110694 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110699 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110705 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110710 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110715 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110720 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110725 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110730 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110735 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110740 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110745 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110752 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110759 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110766 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110771 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110778 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:11:14.113606 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110784 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110789 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110795 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110801 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110806 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110812 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110817 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110822 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110826 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110831 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110835 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110840 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110844 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110849 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110854 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110858 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110864 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110868 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110872 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110877 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:11:14.114376 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110881 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110885 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110889 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110894 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110899 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110906 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110911 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110915 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110920 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110925 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110931 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110936 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110941 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110946 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110951 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110956 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110961 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110966 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110970 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:11:14.115009 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110975 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110979 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110984 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110988 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110993 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.110997 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111002 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111006 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111011 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111015 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111020 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111044 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111048 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111053 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111057 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111066 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111070 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111080 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111084 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111089 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:11:14.115568 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111094 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111098 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111103 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111108 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111118 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111122 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111127 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111131 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111136 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111140 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111146 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111152 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111157 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111161 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111166 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111170 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111176 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111181 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111186 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111868 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:11:14.116127 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111877 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111882 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111887 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111892 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111898 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111902 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111907 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111915 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111920 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111925 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111929 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111936 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111941 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111946 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111951 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111955 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111961 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111966 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111970 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111975 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:11:14.116691 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111979 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111983 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111988 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111993 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.111997 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112002 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112006 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112011 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112018 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112044 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112051 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112056 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112063 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112069 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112074 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112082 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112088 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112093 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112098 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:11:14.117284 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112106 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112111 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112116 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112120 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112125 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112131 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112135 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112140 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112145 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112150 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112157 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112162 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112167 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112172 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112177 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112181 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112186 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112191 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112195 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:11:14.117777 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112200 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112204 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112209 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112214 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112218 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112223 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112227 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112232 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112237 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112241 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112246 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112251 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112255 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112263 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112267 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112272 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112277 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112281 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112285 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112290 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:11:14.118314 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112294 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112301 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112305 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112310 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112316 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112320 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112325 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112330 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112335 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112339 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112343 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112348 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112352 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112356 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112361 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112366 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112370 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112374 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112379 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.112384 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:11:14.118827 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112505 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112529 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112540 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112547 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112554 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112562 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112570 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112577 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112583 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112589 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112595 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112600 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112606 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112611 2559 flags.go:64] FLAG: --cgroup-root="" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112617 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112622 2559 flags.go:64] FLAG: --client-ca-file="" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112627 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112633 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112640 2559 flags.go:64] FLAG: --cluster-domain="" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112645 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112650 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112655 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112661 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112667 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 12:11:14.119364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112673 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112678 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112684 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112690 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112696 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112701 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112706 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112712 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112718 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112724 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112729 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112734 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112739 2559 flags.go:64] FLAG: --enable-server="true" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112745 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112752 2559 flags.go:64] FLAG: --event-burst="100" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112757 2559 flags.go:64] FLAG: --event-qps="50" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112761 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112767 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112772 2559 flags.go:64] FLAG: --eviction-hard="" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112779 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112786 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112791 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112797 2559 flags.go:64] FLAG: --eviction-soft="" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112802 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112808 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 12:11:14.119994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112813 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112820 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112826 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112831 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112836 2559 flags.go:64] FLAG: --feature-gates="" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112842 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112848 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112854 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112859 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112865 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112870 2559 flags.go:64] FLAG: --help="false" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112876 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112881 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112886 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112892 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112898 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112904 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112909 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112915 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112920 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112925 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112931 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112936 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112941 2559 flags.go:64] FLAG: --kube-reserved="" Jul 21 12:11:14.120723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112947 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112952 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112958 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112965 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112970 2559 flags.go:64] FLAG: --lock-file="" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112975 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112981 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112986 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.112996 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113002 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113008 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113013 2559 flags.go:64] FLAG: --logging-format="text" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113019 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113043 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113049 2559 flags.go:64] FLAG: --manifest-url="" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113054 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113062 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113068 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113075 2559 flags.go:64] FLAG: --max-pods="110" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113080 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113085 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113090 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113096 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113101 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113107 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 12:11:14.121469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113112 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113125 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113131 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113136 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113142 2559 flags.go:64] FLAG: --pod-cidr="" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113147 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113157 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113162 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113167 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113172 2559 flags.go:64] FLAG: --port="10250" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113178 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113186 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05a900422bb5bdea5" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113192 2559 flags.go:64] FLAG: --qos-reserved="" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113197 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113202 2559 flags.go:64] FLAG: --register-node="true" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113208 2559 flags.go:64] FLAG: --register-with-taints="" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113214 2559 flags.go:64] FLAG: --registry-burst="10" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113219 2559 flags.go:64] FLAG: --registry-qps="5" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113225 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113231 2559 flags.go:64] FLAG: --reserved-memory="" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113237 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113243 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113249 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113254 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113259 2559 flags.go:64] FLAG: --runonce="false" Jul 21 12:11:14.122673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113263 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113269 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113274 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113280 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113284 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113290 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113295 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113301 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113306 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113311 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113316 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113322 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113327 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113332 2559 flags.go:64] FLAG: --system-cgroups="" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113337 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113347 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113352 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113357 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113366 2559 flags.go:64] FLAG: --tls-min-version="" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113372 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113377 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113382 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113387 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113392 2559 flags.go:64] FLAG: --v="2" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113400 2559 flags.go:64] FLAG: --version="false" Jul 21 12:11:14.123405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113407 2559 flags.go:64] FLAG: --vmodule="" Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113415 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.113421 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113630 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113638 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113644 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113648 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113653 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113659 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113663 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113670 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113677 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113682 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113687 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113692 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113696 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113701 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113708 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113713 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:11:14.124132 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113718 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113723 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113729 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113733 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113738 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113743 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113749 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113754 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113759 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113763 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113768 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113772 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113780 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113786 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113791 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113797 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113802 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113808 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113812 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:11:14.124643 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113817 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113822 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113826 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113832 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113836 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113840 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113845 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113850 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113854 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113859 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113863 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113870 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113874 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113879 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113883 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113888 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113893 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113897 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113902 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113909 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:11:14.125171 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113913 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113918 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113922 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113927 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113932 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113936 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113940 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113945 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113950 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113955 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113960 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113964 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113969 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113974 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113978 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113982 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113987 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113991 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.113996 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114000 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:11:14.125724 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114005 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114010 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114014 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114040 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114046 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114050 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114055 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114060 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114066 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114072 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114077 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114083 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114091 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114098 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114103 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114107 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114112 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114117 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114122 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:11:14.126261 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114126 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114131 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114136 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114141 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.114146 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.114827 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.124013 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.124050 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.124900 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.124913 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.124998 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125005 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125009 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125013 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125017 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:11:14.126755 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125021 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125041 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125047 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125052 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125056 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125059 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125062 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125065 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125069 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125071 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125074 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125077 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125080 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125083 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125088 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125091 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125093 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125096 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125099 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125102 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:11:14.127172 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125105 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125108 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125111 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125114 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125119 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125123 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125128 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125133 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125136 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125139 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125142 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125145 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125148 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125151 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125153 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125158 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125162 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125166 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:11:14.127683 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125169 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125174 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125178 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125182 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125186 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125189 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125193 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125197 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125200 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125204 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125208 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125213 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125217 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125220 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125223 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125225 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125228 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125231 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125234 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125237 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:11:14.128182 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125240 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125243 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125246 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125249 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125252 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125255 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125258 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125261 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125264 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125267 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125270 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125273 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125275 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125280 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125284 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125288 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125294 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125299 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125303 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:11:14.128716 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125306 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125310 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125313 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125316 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125320 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125323 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125326 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125329 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125331 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125334 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125337 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125340 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125343 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125346 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125349 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125352 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:11:14.129233 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125355 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.125361 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125520 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125528 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125532 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125536 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125545 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125550 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125555 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125558 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125561 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125564 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125568 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125572 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 12:11:14.129667 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125576 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125578 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125581 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125584 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125587 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125590 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125593 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125595 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125598 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125601 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125604 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125607 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125609 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125612 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125615 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125618 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125622 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125626 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125631 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125635 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 12:11:14.130047 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125638 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125641 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125644 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125646 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125650 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125652 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125655 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125658 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125661 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125663 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125666 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125669 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125671 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125675 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125678 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125680 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125683 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125686 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125689 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125693 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 12:11:14.130576 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125697 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125699 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125703 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125707 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125712 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125718 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125722 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125726 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125729 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125732 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125736 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125739 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125743 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125746 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125749 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125751 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125754 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125757 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125760 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 12:11:14.131099 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125764 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125768 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125771 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125774 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125776 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125779 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125782 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125787 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125792 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125797 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125802 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125805 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125808 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125811 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125814 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125821 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125823 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125826 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125829 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 12:11:14.131588 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125832 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125835 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125837 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125840 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125843 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125846 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125849 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125852 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:14.125854 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.125860 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 12:11:14.132185 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.126035 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 12:11:14.133076 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.133059 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 12:11:14.134541 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.134528 2559 server.go:1019] "Starting client certificate rotation" Jul 21 12:11:14.134664 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.134642 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 12:11:14.134697 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.134684 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 12:11:14.139276 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.139257 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 12:11:14.140649 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.140620 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 12:11:14.151941 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.151913 2559 log.go:25] "Validated CRI v1 runtime API" Jul 21 12:11:14.154106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.154087 2559 log.go:25] "Validated CRI v1 image API" Jul 21 12:11:14.155017 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.154999 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 12:11:14.158882 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.158858 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bca70aff-9b09-49a1-8a64-f55ef83158b8:/dev/nvme0n1p3 d0dc627a-7803-4d20-ba46-1faab4244041:/dev/nvme0n1p4] Jul 21 12:11:14.158946 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.158882 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 12:11:14.163323 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.163291 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 12:11:14.167571 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.167434 2559 manager.go:216] Machine: {Timestamp:2026-07-21 12:11:14.165403535 +0000 UTC m=+0.334754807 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097384 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ed3384c928e0c56d96c66b7bfdd2e SystemUUID:ec2ed338-4c92-8e0c-56d9-6c66b7bfdd2e BootID:7344ee06-76be-4fb0-9a38-9505344e864d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:b7:09:5a:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:b7:09:5a:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:db:eb:3e:2b:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 12:11:14.167571 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.167561 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 12:11:14.167750 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.167684 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 12:11:14.170819 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.170785 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 12:11:14.171001 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.170823 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 12:11:14.171668 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.171653 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 12:11:14.171726 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.171674 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 12:11:14.171726 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.171686 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 12:11:14.173088 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.173074 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 12:11:14.174311 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.174296 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 12:11:14.178319 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.178301 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 21 12:11:14.178535 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.178521 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 12:11:14.178597 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.178565 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 21 12:11:14.178597 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.178582 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 12:11:14.178690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.178604 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 12:11:14.178690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.178619 2559 kubelet.go:398] "Adding apiserver pod source" Jul 21 12:11:14.178690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.178635 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 12:11:14.179092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.179063 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 12:11:14.179092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.179086 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 12:11:14.181541 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.181524 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 12:11:14.183427 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.183411 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 12:11:14.183489 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.183445 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 12:11:14.184473 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184461 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 12:11:14.184518 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184478 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 12:11:14.184518 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184485 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 12:11:14.184518 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184491 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 12:11:14.184518 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184500 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 12:11:14.184518 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184506 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 12:11:14.184667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184611 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 12:11:14.184667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184618 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 12:11:14.184667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184626 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 12:11:14.184667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184633 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 12:11:14.184667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184645 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 12:11:14.184667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.184654 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 12:11:14.185388 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.185376 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 12:11:14.185388 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.185389 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 12:11:14.188633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.188602 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 12:11:14.188633 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.188606 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 12:11:14.188779 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.188684 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 12:11:14.189568 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.189552 2559 server.go:1268] "Started kubelet" Jul 21 12:11:14.189673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.189637 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 12:11:14.189843 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.189691 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 12:11:14.189903 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.189888 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 12:11:14.190593 ip-10-0-135-97 systemd[1]: Started Kubernetes Kubelet. Jul 21 12:11:14.191774 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.191654 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 12:11:14.192548 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.192518 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jvwjf" Jul 21 12:11:14.193338 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.193322 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 21 12:11:14.195421 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.195397 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 12:11:14.195594 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.194606 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-97.ec2.internal.18c44cdf467c3314 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-97.ec2.internal,UID:ip-10-0-135-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-97.ec2.internal,},FirstTimestamp:2026-07-21 12:11:14.189521684 +0000 UTC m=+0.358872989,LastTimestamp:2026-07-21 12:11:14.189521684 +0000 UTC m=+0.358872989,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-97.ec2.internal,}" Jul 21 12:11:14.198144 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.198122 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 12:11:14.198144 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.198132 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 12:11:14.198325 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.198215 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.198706 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.198686 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 12:11:14.198796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.198718 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 12:11:14.198796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.198730 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 12:11:14.198941 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.198925 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 21 12:11:14.198941 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.198941 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 21 12:11:14.199221 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.199202 2559 factory.go:55] Registering systemd factory Jul 21 12:11:14.199302 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.199225 2559 factory.go:223] Registration of the systemd container factory successfully Jul 21 12:11:14.199474 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.199456 2559 factory.go:153] Registering CRI-O factory Jul 21 12:11:14.199474 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.199475 2559 factory.go:223] Registration of the crio container factory successfully Jul 21 12:11:14.199639 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.199535 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 12:11:14.199639 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.199562 2559 factory.go:103] Registering Raw factory Jul 21 12:11:14.199639 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.199578 2559 manager.go:1196] Started watching for new ooms in manager Jul 21 12:11:14.200119 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.200102 2559 manager.go:349] Starting recovery of all containers Jul 21 12:11:14.201796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.201771 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jvwjf" Jul 21 12:11:14.206293 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.206259 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 12:11:14.206403 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.206005 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 12:11:14.214505 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.214484 2559 manager.go:354] Recovery completed Jul 21 12:11:14.218769 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.218748 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:11:14.220776 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.220757 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:11:14.220848 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.220794 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:11:14.220848 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.220810 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:11:14.221412 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.221397 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 12:11:14.221465 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.221411 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 12:11:14.221465 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.221415 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:11:14.221465 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.221429 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 21 12:11:14.222203 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.222187 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:11:14.222289 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.222240 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:11:14.222289 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.222276 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:11:14.222871 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.222767 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-97.ec2.internal.18c44cdf48591a63 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-97.ec2.internal,UID:ip-10-0-135-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-97.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-97.ec2.internal,},FirstTimestamp:2026-07-21 12:11:14.220776035 +0000 UTC m=+0.390127311,LastTimestamp:2026-07-21 12:11:14.220776035 +0000 UTC m=+0.390127311,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-97.ec2.internal,}" Jul 21 12:11:14.223586 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.223573 2559 policy_none.go:49] "None policy: Start" Jul 21 12:11:14.223630 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.223590 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 12:11:14.223630 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.223601 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 12:11:14.225672 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.225661 2559 policy_none.go:47] "Start" Jul 21 12:11:14.262967 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.262729 2559 manager.go:337] "Starting Device Plugin manager" Jul 21 12:11:14.263097 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.263007 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 12:11:14.263097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.263020 2559 server.go:87] "Starting device plugin registration server" Jul 21 12:11:14.263295 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.263283 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 12:11:14.263345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.263297 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 12:11:14.263437 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.263420 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 12:11:14.263531 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.263517 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 12:11:14.263580 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.263533 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 12:11:14.264277 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.264254 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 12:11:14.264413 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.264313 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.350387 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.350331 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 12:11:14.352345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.351695 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 12:11:14.352345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.351736 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 12:11:14.352345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.351761 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 12:11:14.352345 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.351798 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 12:11:14.354512 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.354485 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:11:14.363842 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.363822 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:11:14.365098 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.365081 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:11:14.365186 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.365117 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:11:14.365186 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.365128 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:11:14.365186 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.365153 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.373191 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.373169 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.373320 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.373196 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-97.ec2.internal\": node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.389303 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.389273 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.454062 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.454015 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal"] Jul 21 12:11:14.454229 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.454110 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:11:14.455178 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.455159 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:11:14.455251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.455197 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:11:14.455251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.455209 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:11:14.457196 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.457178 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:11:14.457334 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.457317 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.457396 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.457357 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:11:14.457999 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.457985 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:11:14.458065 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.458016 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:11:14.458065 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.458029 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:11:14.458136 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.457985 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:11:14.458136 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.458093 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:11:14.458136 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.458103 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:11:14.459964 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.459934 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.460081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.459987 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 12:11:14.460661 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.460646 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 12:11:14.460731 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.460667 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 12:11:14.460731 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.460680 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 12:11:14.483011 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.482987 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-97.ec2.internal\" not found" node="ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.487803 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.487783 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-97.ec2.internal\" not found" node="ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.490003 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.489975 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.500356 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.500325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6fa6df74d0193c0424d08a53679b0d40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal\" (UID: \"6fa6df74d0193c0424d08a53679b0d40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.500480 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.500357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fa6df74d0193c0424d08a53679b0d40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal\" (UID: \"6fa6df74d0193c0424d08a53679b0d40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.500480 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.500392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e306b331cd25c79c60bbf3257cbc2b52-config\") pod \"kube-apiserver-proxy-ip-10-0-135-97.ec2.internal\" (UID: \"e306b331cd25c79c60bbf3257cbc2b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.590370 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.590338 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.600670 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.600597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6fa6df74d0193c0424d08a53679b0d40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal\" (UID: \"6fa6df74d0193c0424d08a53679b0d40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.600670 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.600630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fa6df74d0193c0424d08a53679b0d40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal\" (UID: \"6fa6df74d0193c0424d08a53679b0d40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.600670 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.600648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e306b331cd25c79c60bbf3257cbc2b52-config\") pod \"kube-apiserver-proxy-ip-10-0-135-97.ec2.internal\" (UID: \"e306b331cd25c79c60bbf3257cbc2b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.600893 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.600726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6fa6df74d0193c0424d08a53679b0d40-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal\" (UID: \"6fa6df74d0193c0424d08a53679b0d40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.600893 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.600786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e306b331cd25c79c60bbf3257cbc2b52-config\") pod \"kube-apiserver-proxy-ip-10-0-135-97.ec2.internal\" (UID: \"e306b331cd25c79c60bbf3257cbc2b52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.600893 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.600821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fa6df74d0193c0424d08a53679b0d40-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal\" (UID: \"6fa6df74d0193c0424d08a53679b0d40\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.691412 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.691365 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.792609 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.792567 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.792749 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.792657 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.792749 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:14.792717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:14.894068 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.893973 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:14.999779 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:14.999737 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-97.ec2.internal\" not found" Jul 21 12:11:15.006940 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.006920 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:11:15.099240 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.099202 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" Jul 21 12:11:15.111829 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.111796 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 12:11:15.113668 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.113651 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" Jul 21 12:11:15.123795 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.123767 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 12:11:15.134180 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.134139 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 12:11:15.134368 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.134340 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 12:11:15.134412 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.134340 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 12:11:15.159236 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.159161 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:11:15.179196 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.179162 2559 apiserver.go:52] "Watching apiserver" Jul 21 12:11:15.185786 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.185759 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 12:11:15.187856 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.187830 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jqth5","openshift-network-diagnostics/network-check-target-nf86x","openshift-network-operator/iptables-alerter-72rzb","openshift-ovn-kubernetes/ovnkube-node-7qs5s","kube-system/global-pull-secret-syncer-xmpjs","kube-system/konnectivity-agent-9jkd6","kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl","openshift-cluster-node-tuning-operator/tuned-94lz5","openshift-dns/node-resolver-j5tm5","openshift-image-registry/node-ca-bmb77","openshift-multus/multus-additional-cni-plugins-74znp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal","openshift-multus/multus-zpmfn"] Jul 21 12:11:15.191777 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.191754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:15.191873 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.191829 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:15.194120 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.194104 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:15.194195 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.194172 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:15.196692 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.196675 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.198490 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.198472 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 12:11:15.199160 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.198956 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 12:11:15.199160 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.199013 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 12:11:15.199160 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.199042 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:11:15.199160 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.199045 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cwkrs\"" Jul 21 12:11:15.199455 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.199437 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.201734 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.201706 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 12:11:15.201734 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.201733 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 12:11:15.201985 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.201866 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6b6b4\"" Jul 21 12:11:15.201985 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.201929 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 12:11:15.202114 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.202099 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 12:11:15.202114 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.202105 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 12:11:15.202225 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.202147 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 12:11:15.202225 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.202189 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.202318 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.202295 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:15.203254 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-cni-bin\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203357 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:15.203357 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-kubelet\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203357 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-systemd-units\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203357 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203344 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-var-lib-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83a7f9b9-ee96-4874-bd0d-fda250113e12-host-slash\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.203559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnvg\" (UniqueName: \"kubernetes.io/projected/83a7f9b9-ee96-4874-bd0d-fda250113e12-kube-api-access-xvnvg\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.203559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203461 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-systemd\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203489 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-cni-netd\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203513 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-env-overrides\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83a7f9b9-ee96-4874-bd0d-fda250113e12-iptables-alerter-script\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203592 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovn-node-metrics-cert\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovnkube-script-lib\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203688 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7hp\" (UniqueName: \"kubernetes.io/projected/eef530bc-aba7-4b53-93d6-c8818dcffe5a-kube-api-access-jj7hp\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-slash\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-etc-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-node-log\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-log-socket\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.203867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.204309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-run-netns\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.204309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203934 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-ovn\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.204309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.203959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovnkube-config\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.204309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.204004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9fm\" (UniqueName: \"kubernetes.io/projected/c5f44d8c-0fac-49cf-8e58-e8e33a638319-kube-api-access-wp9fm\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:15.204887 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.204871 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.206939 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.206918 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t8dm4\"" Jul 21 12:11:15.207073 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.207036 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 12:11:15.207156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.207092 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 12:11:15.207287 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.207261 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 12:06:15 +0000 UTC" deadline="2027-12-21 11:12:50.80530764 +0000 UTC" Jul 21 12:11:15.207341 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.207288 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12431h1m35.598023195s" Jul 21 12:11:15.208118 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.208103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.211013 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.210405 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 12:11:15.211013 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.210421 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 12:11:15.211013 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.210437 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87qfx\"" Jul 21 12:11:15.211270 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.211036 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 12:11:15.212350 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.211634 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.214386 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.214349 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 12:11:15.214386 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.214358 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:11:15.214678 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.214659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.216069 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.216049 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kcwn9\"" Jul 21 12:11:15.216276 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.216168 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 12:11:15.216904 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.216883 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 12:11:15.217192 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.217173 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kl9lt\"" Jul 21 12:11:15.217344 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.217328 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 12:11:15.217689 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.217660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.220127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.220107 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4572f\"" Jul 21 12:11:15.220256 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.220132 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 12:11:15.220256 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.220157 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 12:11:15.220256 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.220114 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 12:11:15.220412 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.220389 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.222882 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.222860 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 12:11:15.223014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.222885 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 12:11:15.223014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.222861 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 12:11:15.223014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.222933 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z2tz5\"" Jul 21 12:11:15.223014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.222957 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.223446 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.223354 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 12:11:15.223446 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.223415 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 12:11:15.225299 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.225276 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j8dgf\"" Jul 21 12:11:15.225381 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.225281 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 12:11:15.237465 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.237443 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mn7m2" Jul 21 12:11:15.247721 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.247699 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mn7m2" Jul 21 12:11:15.299607 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.299583 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 12:11:15.304369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304348 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-socket-dir-parent\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.304458 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83a7f9b9-ee96-4874-bd0d-fda250113e12-iptables-alerter-script\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.304458 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-os-release\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.304458 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-lib-modules\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.304633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-var-lib-kubelet\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.304633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-registration-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.304633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-sys-fs\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.304633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.304633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:15.304633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.304633 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-slash\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-run-netns\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9fm\" (UniqueName: \"kubernetes.io/projected/c5f44d8c-0fac-49cf-8e58-e8e33a638319-kube-api-access-wp9fm\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-modprobe-d\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-kubernetes\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-run-netns\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-slash\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304769 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysctl-conf\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysconfig\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wvw\" (UniqueName: \"kubernetes.io/projected/206118a1-01d3-44ca-894f-0c229a671f20-kube-api-access-v6wvw\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304893 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-serviceca\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/426a4044-78b3-421b-abd3-85f56961434a-dbus\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304946 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eae41061-6724-4683-b038-5495c849fa21-agent-certs\") pod \"konnectivity-agent-9jkd6\" (UID: \"eae41061-6724-4683-b038-5495c849fa21\") " pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.304995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.304990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysctl-d\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eae41061-6724-4683-b038-5495c849fa21-konnectivity-ca\") pod \"konnectivity-agent-9jkd6\" (UID: \"eae41061-6724-4683-b038-5495c849fa21\") " pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305111 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83a7f9b9-ee96-4874-bd0d-fda250113e12-iptables-alerter-script\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305120 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-systemd-units\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnvg\" (UniqueName: \"kubernetes.io/projected/83a7f9b9-ee96-4874-bd0d-fda250113e12-kube-api-access-xvnvg\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-systemd-units\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-cni-binary-copy\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-host\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305388 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfb7\" (UniqueName: \"kubernetes.io/projected/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-kube-api-access-clfb7\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-systemd\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-cni-netd\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-env-overrides\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305540 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-netns\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305542 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-systemd\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305549 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-cni-netd\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-hostroot\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.305576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-sys\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305622 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-tuned\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c23a5fec-2f36-44cd-811f-2e5ba9534759-tmp\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305720 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovn-node-metrics-cert\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovnkube-script-lib\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-k8s-cni-cncf-io\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-kubelet\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-run\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-host\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7hp\" (UniqueName: \"kubernetes.io/projected/eef530bc-aba7-4b53-93d6-c8818dcffe5a-kube-api-access-jj7hp\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305928 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-etc-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.305994 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-etc-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-node-log\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-env-overrides\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-log-socket\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.306395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306116 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-node-log\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-log-socket\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-cnibin\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306149 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-conf-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-ovn\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovnkube-config\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-daemon-config\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306316 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovnkube-script-lib\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-multus-certs\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306357 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-run-ovn\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9ms\" (UniqueName: \"kubernetes.io/projected/14cd5476-2b2c-4e0b-bf29-78901a796492-kube-api-access-sj9ms\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxl7\" (UniqueName: \"kubernetes.io/projected/c23a5fec-2f36-44cd-811f-2e5ba9534759-kube-api-access-9fxl7\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206118a1-01d3-44ca-894f-0c229a671f20-tmp-dir\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.307284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovnkube-config\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/426a4044-78b3-421b-abd3-85f56961434a-kubelet-config\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-system-cni-dir\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306968 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-os-release\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.306985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-cni-multus\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-etc-kubernetes\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-systemd\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/206118a1-01d3-44ca-894f-0c229a671f20-hosts-file\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307125 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-device-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307196 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cp6\" (UniqueName: \"kubernetes.io/projected/08891ba6-492f-4dfd-a54c-b826dcf02d92-kube-api-access-d8cp6\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307215 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14cd5476-2b2c-4e0b-bf29-78901a796492-cni-binary-copy\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307234 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-cni-bin\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-socket-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.308156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-cni-bin\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307349 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-cni-bin\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-kubelet\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307403 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt28p\" (UniqueName: \"kubernetes.io/projected/9939f893-b6d8-4c95-aef4-dbe6866eaabe-kube-api-access-zt28p\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.307435 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-var-lib-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83a7f9b9-ee96-4874-bd0d-fda250113e12-host-slash\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-host-kubelet\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.307519 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs podName:c5f44d8c-0fac-49cf-8e58-e8e33a638319 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:15.807482196 +0000 UTC m=+1.976833473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs") pod "network-metrics-daemon-jqth5" (UID: "c5f44d8c-0fac-49cf-8e58-e8e33a638319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307513 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eef530bc-aba7-4b53-93d6-c8818dcffe5a-var-lib-openvswitch\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83a7f9b9-ee96-4874-bd0d-fda250113e12-host-slash\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-system-cni-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307564 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-cni-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.308699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.307585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-cnibin\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.309886 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.309865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eef530bc-aba7-4b53-93d6-c8818dcffe5a-ovn-node-metrics-cert\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.313489 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.313464 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:11:15.313489 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.313489 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:11:15.313668 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.313500 2559 projected.go:196] Error preparing data for projected volume kube-api-access-fhzhr for pod openshift-network-diagnostics/network-check-target-nf86x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:15.313668 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.313567 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr podName:9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d nodeName:}" failed. No retries permitted until 2026-07-21 12:11:15.813546332 +0000 UTC m=+1.982897590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fhzhr" (UniqueName: "kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr") pod "network-check-target-nf86x" (UID: "9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:15.316247 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.316222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9fm\" (UniqueName: \"kubernetes.io/projected/c5f44d8c-0fac-49cf-8e58-e8e33a638319-kube-api-access-wp9fm\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:15.316394 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.316223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnvg\" (UniqueName: \"kubernetes.io/projected/83a7f9b9-ee96-4874-bd0d-fda250113e12-kube-api-access-xvnvg\") pod \"iptables-alerter-72rzb\" (UID: \"83a7f9b9-ee96-4874-bd0d-fda250113e12\") " pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.316490 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.316466 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7hp\" (UniqueName: \"kubernetes.io/projected/eef530bc-aba7-4b53-93d6-c8818dcffe5a-kube-api-access-jj7hp\") pod \"ovnkube-node-7qs5s\" (UID: \"eef530bc-aba7-4b53-93d6-c8818dcffe5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.351850 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.351818 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa6df74d0193c0424d08a53679b0d40.slice/crio-74c17ff94e9b9c94ee2b9f560e3d2383b4adac3d6fb8ec36c107aca1fdec1479 WatchSource:0}: Error finding container 74c17ff94e9b9c94ee2b9f560e3d2383b4adac3d6fb8ec36c107aca1fdec1479: Status 404 returned error can't find the container with id 74c17ff94e9b9c94ee2b9f560e3d2383b4adac3d6fb8ec36c107aca1fdec1479 Jul 21 12:11:15.355155 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.355129 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode306b331cd25c79c60bbf3257cbc2b52.slice/crio-af02c5e20ae2525fbd8af83bce97e7b51d9043ef66ae8054dd784d1e95c47f49 WatchSource:0}: Error finding container af02c5e20ae2525fbd8af83bce97e7b51d9043ef66ae8054dd784d1e95c47f49: Status 404 returned error can't find the container with id af02c5e20ae2525fbd8af83bce97e7b51d9043ef66ae8054dd784d1e95c47f49 Jul 21 12:11:15.357846 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.357829 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:11:15.407847 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.407811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.407847 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.407845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt28p\" (UniqueName: \"kubernetes.io/projected/9939f893-b6d8-4c95-aef4-dbe6866eaabe-kube-api-access-zt28p\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.407878 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-system-cni-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.407898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-cni-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-cnibin\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408022 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-socket-dir-parent\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-os-release\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-lib-modules\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-var-lib-kubelet\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-system-cni-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-registration-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.408127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-cnibin\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-socket-dir-parent\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-sys-fs\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-var-lib-kubelet\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408202 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-sys-fs\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-modprobe-d\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-registration-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-lib-modules\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-kubernetes\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408292 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-cni-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-modprobe-d\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysctl-conf\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysconfig\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-kubernetes\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wvw\" (UniqueName: \"kubernetes.io/projected/206118a1-01d3-44ca-894f-0c229a671f20-kube-api-access-v6wvw\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-serviceca\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysconfig\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/426a4044-78b3-421b-abd3-85f56961434a-dbus\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.408612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408428 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysctl-conf\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-os-release\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408451 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eae41061-6724-4683-b038-5495c849fa21-agent-certs\") pod \"konnectivity-agent-9jkd6\" (UID: \"eae41061-6724-4683-b038-5495c849fa21\") " pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysctl-d\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408493 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/426a4044-78b3-421b-abd3-85f56961434a-dbus\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eae41061-6724-4683-b038-5495c849fa21-konnectivity-ca\") pod \"konnectivity-agent-9jkd6\" (UID: \"eae41061-6724-4683-b038-5495c849fa21\") " pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-cni-binary-copy\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408558 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-host\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408611 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfb7\" (UniqueName: \"kubernetes.io/projected/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-kube-api-access-clfb7\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-sysctl-d\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-host\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-netns\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-hostroot\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408714 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-netns\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-sys\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-tuned\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.409514 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c23a5fec-2f36-44cd-811f-2e5ba9534759-tmp\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-k8s-cni-cncf-io\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408830 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-serviceca\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-k8s-cni-cncf-io\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-kubelet\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-kubelet\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-hostroot\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-sys\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.408996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-run\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-host\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409047 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-cnibin\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-conf-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-daemon-config\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-cni-binary-copy\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-multus-certs\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9ms\" (UniqueName: \"kubernetes.io/projected/14cd5476-2b2c-4e0b-bf29-78901a796492-kube-api-access-sj9ms\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.410449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eae41061-6724-4683-b038-5495c849fa21-konnectivity-ca\") pod \"konnectivity-agent-9jkd6\" (UID: \"eae41061-6724-4683-b038-5495c849fa21\") " pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-cnibin\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-run\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-host\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxl7\" (UniqueName: \"kubernetes.io/projected/c23a5fec-2f36-44cd-811f-2e5ba9534759-kube-api-access-9fxl7\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409322 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206118a1-01d3-44ca-894f-0c229a671f20-tmp-dir\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/426a4044-78b3-421b-abd3-85f56961434a-kubelet-config\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-system-cni-dir\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409414 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-run-multus-certs\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-os-release\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409466 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-conf-dir\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-cni-multus\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409517 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08891ba6-492f-4dfd-a54c-b826dcf02d92-system-cni-dir\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-etc-kubernetes\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-systemd\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/426a4044-78b3-421b-abd3-85f56961434a-kubelet-config\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/206118a1-01d3-44ca-894f-0c229a671f20-hosts-file\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.411117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-device-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-etc-kubernetes\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/206118a1-01d3-44ca-894f-0c229a671f20-hosts-file\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-cni-multus\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14cd5476-2b2c-4e0b-bf29-78901a796492-multus-daemon-config\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409836 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-systemd\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-os-release\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-device-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cp6\" (UniqueName: \"kubernetes.io/projected/08891ba6-492f-4dfd-a54c-b826dcf02d92-kube-api-access-d8cp6\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.409990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14cd5476-2b2c-4e0b-bf29-78901a796492-cni-binary-copy\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-cni-bin\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410239 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14cd5476-2b2c-4e0b-bf29-78901a796492-host-var-lib-cni-bin\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206118a1-01d3-44ca-894f-0c229a671f20-tmp-dir\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-socket-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.411770 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.410390 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:15.412369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08891ba6-492f-4dfd-a54c-b826dcf02d92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.412369 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.410460 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret podName:426a4044-78b3-421b-abd3-85f56961434a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:15.9104414 +0000 UTC m=+2.079792675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret") pod "global-pull-secret-syncer-xmpjs" (UID: "426a4044-78b3-421b-abd3-85f56961434a") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:15.412369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9939f893-b6d8-4c95-aef4-dbe6866eaabe-socket-dir\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.412369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.410597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14cd5476-2b2c-4e0b-bf29-78901a796492-cni-binary-copy\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.412369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.411638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c23a5fec-2f36-44cd-811f-2e5ba9534759-etc-tuned\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.412369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.411732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c23a5fec-2f36-44cd-811f-2e5ba9534759-tmp\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.412369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.411837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eae41061-6724-4683-b038-5495c849fa21-agent-certs\") pod \"konnectivity-agent-9jkd6\" (UID: \"eae41061-6724-4683-b038-5495c849fa21\") " pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.416824 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.416786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfb7\" (UniqueName: \"kubernetes.io/projected/2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d-kube-api-access-clfb7\") pod \"node-ca-bmb77\" (UID: \"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d\") " pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.416999 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.416979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wvw\" (UniqueName: \"kubernetes.io/projected/206118a1-01d3-44ca-894f-0c229a671f20-kube-api-access-v6wvw\") pod \"node-resolver-j5tm5\" (UID: \"206118a1-01d3-44ca-894f-0c229a671f20\") " pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.417271 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.417248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt28p\" (UniqueName: \"kubernetes.io/projected/9939f893-b6d8-4c95-aef4-dbe6866eaabe-kube-api-access-zt28p\") pod \"aws-ebs-csi-driver-node-hj9fl\" (UID: \"9939f893-b6d8-4c95-aef4-dbe6866eaabe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.417386 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.417371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxl7\" (UniqueName: \"kubernetes.io/projected/c23a5fec-2f36-44cd-811f-2e5ba9534759-kube-api-access-9fxl7\") pod \"tuned-94lz5\" (UID: \"c23a5fec-2f36-44cd-811f-2e5ba9534759\") " pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.417805 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.417787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9ms\" (UniqueName: \"kubernetes.io/projected/14cd5476-2b2c-4e0b-bf29-78901a796492-kube-api-access-sj9ms\") pod \"multus-zpmfn\" (UID: \"14cd5476-2b2c-4e0b-bf29-78901a796492\") " pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.420836 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.420815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cp6\" (UniqueName: \"kubernetes.io/projected/08891ba6-492f-4dfd-a54c-b826dcf02d92-kube-api-access-d8cp6\") pod \"multus-additional-cni-plugins-74znp\" (UID: \"08891ba6-492f-4dfd-a54c-b826dcf02d92\") " pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.519242 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.519204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-72rzb" Jul 21 12:11:15.530334 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.530299 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a7f9b9_ee96_4874_bd0d_fda250113e12.slice/crio-5d849aacda89c21e77a031b81e40bffc518965c1d6ee2686de3bb7e4cbd9f5e7 WatchSource:0}: Error finding container 5d849aacda89c21e77a031b81e40bffc518965c1d6ee2686de3bb7e4cbd9f5e7: Status 404 returned error can't find the container with id 5d849aacda89c21e77a031b81e40bffc518965c1d6ee2686de3bb7e4cbd9f5e7 Jul 21 12:11:15.532551 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.532532 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:15.536611 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.536593 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:15.540588 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.540563 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" Jul 21 12:11:15.543052 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.543022 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef530bc_aba7_4b53_93d6_c8818dcffe5a.slice/crio-15a545d1713251cad782637348f4a6787b08d6fce88c005596757098e9fc6911 WatchSource:0}: Error finding container 15a545d1713251cad782637348f4a6787b08d6fce88c005596757098e9fc6911: Status 404 returned error can't find the container with id 15a545d1713251cad782637348f4a6787b08d6fce88c005596757098e9fc6911 Jul 21 12:11:15.547420 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.547398 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae41061_6724_4683_b038_5495c849fa21.slice/crio-04d00c8bac9679c07ce2479fa933cc15c1532fa953034502a14d030093d8ed77 WatchSource:0}: Error finding container 04d00c8bac9679c07ce2479fa933cc15c1532fa953034502a14d030093d8ed77: Status 404 returned error can't find the container with id 04d00c8bac9679c07ce2479fa933cc15c1532fa953034502a14d030093d8ed77 Jul 21 12:11:15.551279 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.551255 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9939f893_b6d8_4c95_aef4_dbe6866eaabe.slice/crio-173eff428b14dfbaad095cb21184cb37cf63c1febd05611320055dd0ae3fa43f WatchSource:0}: Error finding container 173eff428b14dfbaad095cb21184cb37cf63c1febd05611320055dd0ae3fa43f: Status 404 returned error can't find the container with id 173eff428b14dfbaad095cb21184cb37cf63c1febd05611320055dd0ae3fa43f Jul 21 12:11:15.562311 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.562272 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-94lz5" Jul 21 12:11:15.573827 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.573798 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23a5fec_2f36_44cd_811f_2e5ba9534759.slice/crio-1286f752f81985540dfefc94f421c16b99706e17cb61bec8a0b6877054a75529 WatchSource:0}: Error finding container 1286f752f81985540dfefc94f421c16b99706e17cb61bec8a0b6877054a75529: Status 404 returned error can't find the container with id 1286f752f81985540dfefc94f421c16b99706e17cb61bec8a0b6877054a75529 Jul 21 12:11:15.577618 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.577599 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:11:15.578445 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.578427 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j5tm5" Jul 21 12:11:15.587095 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.587064 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206118a1_01d3_44ca_894f_0c229a671f20.slice/crio-4b89111d85bbde282565dd21850b328891a0ed795da9341b0c621a4f7380f370 WatchSource:0}: Error finding container 4b89111d85bbde282565dd21850b328891a0ed795da9341b0c621a4f7380f370: Status 404 returned error can't find the container with id 4b89111d85bbde282565dd21850b328891a0ed795da9341b0c621a4f7380f370 Jul 21 12:11:15.593291 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.593269 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmb77" Jul 21 12:11:15.599577 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.599492 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74znp" Jul 21 12:11:15.601848 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.601822 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1f89f7_ac5a_4452_8a2f_1e74a36c5e8d.slice/crio-7c12b989a8601fe38837ceacb64655340b3d36ec6da7d8c7dc8bdcbd02ceb91b WatchSource:0}: Error finding container 7c12b989a8601fe38837ceacb64655340b3d36ec6da7d8c7dc8bdcbd02ceb91b: Status 404 returned error can't find the container with id 7c12b989a8601fe38837ceacb64655340b3d36ec6da7d8c7dc8bdcbd02ceb91b Jul 21 12:11:15.608892 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.608869 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08891ba6_492f_4dfd_a54c_b826dcf02d92.slice/crio-5200f3800099748e0649ebe73a5f6284d6c5fedc7a76ed3179097254e61552d0 WatchSource:0}: Error finding container 5200f3800099748e0649ebe73a5f6284d6c5fedc7a76ed3179097254e61552d0: Status 404 returned error can't find the container with id 5200f3800099748e0649ebe73a5f6284d6c5fedc7a76ed3179097254e61552d0 Jul 21 12:11:15.616162 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.616138 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zpmfn" Jul 21 12:11:15.626465 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:15.626436 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cd5476_2b2c_4e0b_bf29_78901a796492.slice/crio-71d8e946f50d5e14d5b56e1b31880da865a39d10a99df9953eca2a02816f8547 WatchSource:0}: Error finding container 71d8e946f50d5e14d5b56e1b31880da865a39d10a99df9953eca2a02816f8547: Status 404 returned error can't find the container with id 71d8e946f50d5e14d5b56e1b31880da865a39d10a99df9953eca2a02816f8547 Jul 21 12:11:15.813519 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.813413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:15.813687 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.813582 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:15.813687 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.813651 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs podName:c5f44d8c-0fac-49cf-8e58-e8e33a638319 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:16.813634426 +0000 UTC m=+2.982985687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs") pod "network-metrics-daemon-jqth5" (UID: "c5f44d8c-0fac-49cf-8e58-e8e33a638319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:15.914466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.914208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:15.914466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:15.914285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:15.914466 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.914437 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:15.914744 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.914444 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:11:15.914744 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.914532 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret podName:426a4044-78b3-421b-abd3-85f56961434a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:16.914510403 +0000 UTC m=+3.083861659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret") pod "global-pull-secret-syncer-xmpjs" (UID: "426a4044-78b3-421b-abd3-85f56961434a") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:15.914744 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.914544 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:11:15.914744 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.914564 2559 projected.go:196] Error preparing data for projected volume kube-api-access-fhzhr for pod openshift-network-diagnostics/network-check-target-nf86x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:15.914744 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:15.914622 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr podName:9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d nodeName:}" failed. No retries permitted until 2026-07-21 12:11:16.914603435 +0000 UTC m=+3.083954709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhzhr" (UniqueName: "kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr") pod "network-check-target-nf86x" (UID: "9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:16.251629 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.251498 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 12:06:15 +0000 UTC" deadline="2028-03-21 15:47:59.556740366 +0000 UTC" Jul 21 12:11:16.251629 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.251572 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14619h36m43.305172483s" Jul 21 12:11:16.353666 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.353628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:16.353869 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.353790 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:16.354609 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.354322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:16.354609 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.354433 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:16.370648 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.370582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" event={"ID":"9939f893-b6d8-4c95-aef4-dbe6866eaabe","Type":"ContainerStarted","Data":"173eff428b14dfbaad095cb21184cb37cf63c1febd05611320055dd0ae3fa43f"} Jul 21 12:11:16.382326 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.382224 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jkd6" event={"ID":"eae41061-6724-4683-b038-5495c849fa21","Type":"ContainerStarted","Data":"04d00c8bac9679c07ce2479fa933cc15c1532fa953034502a14d030093d8ed77"} Jul 21 12:11:16.385179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.385054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"15a545d1713251cad782637348f4a6787b08d6fce88c005596757098e9fc6911"} Jul 21 12:11:16.395793 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.395717 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-72rzb" event={"ID":"83a7f9b9-ee96-4874-bd0d-fda250113e12","Type":"ContainerStarted","Data":"5d849aacda89c21e77a031b81e40bffc518965c1d6ee2686de3bb7e4cbd9f5e7"} Jul 21 12:11:16.397688 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.397610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" event={"ID":"e306b331cd25c79c60bbf3257cbc2b52","Type":"ContainerStarted","Data":"af02c5e20ae2525fbd8af83bce97e7b51d9043ef66ae8054dd784d1e95c47f49"} Jul 21 12:11:16.402275 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.402237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerStarted","Data":"5200f3800099748e0649ebe73a5f6284d6c5fedc7a76ed3179097254e61552d0"} Jul 21 12:11:16.408322 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.408256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmb77" event={"ID":"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d","Type":"ContainerStarted","Data":"7c12b989a8601fe38837ceacb64655340b3d36ec6da7d8c7dc8bdcbd02ceb91b"} Jul 21 12:11:16.416365 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.416332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" event={"ID":"6fa6df74d0193c0424d08a53679b0d40","Type":"ContainerStarted","Data":"74c17ff94e9b9c94ee2b9f560e3d2383b4adac3d6fb8ec36c107aca1fdec1479"} Jul 21 12:11:16.423686 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.423641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zpmfn" event={"ID":"14cd5476-2b2c-4e0b-bf29-78901a796492","Type":"ContainerStarted","Data":"71d8e946f50d5e14d5b56e1b31880da865a39d10a99df9953eca2a02816f8547"} Jul 21 12:11:16.427808 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.427715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j5tm5" event={"ID":"206118a1-01d3-44ca-894f-0c229a671f20","Type":"ContainerStarted","Data":"4b89111d85bbde282565dd21850b328891a0ed795da9341b0c621a4f7380f370"} Jul 21 12:11:16.429895 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.429826 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-94lz5" event={"ID":"c23a5fec-2f36-44cd-811f-2e5ba9534759","Type":"ContainerStarted","Data":"1286f752f81985540dfefc94f421c16b99706e17cb61bec8a0b6877054a75529"} Jul 21 12:11:16.679587 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.679480 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:11:16.810128 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.810095 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 12:11:16.822755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.822714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:16.822942 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.822893 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:16.823039 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.822961 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs podName:c5f44d8c-0fac-49cf-8e58-e8e33a638319 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:18.82294103 +0000 UTC m=+4.992292289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs") pod "network-metrics-daemon-jqth5" (UID: "c5f44d8c-0fac-49cf-8e58-e8e33a638319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:16.924189 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.923369 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:16.924189 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:16.923458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:16.924189 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.923663 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:11:16.924189 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.923686 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:11:16.924189 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.923701 2559 projected.go:196] Error preparing data for projected volume kube-api-access-fhzhr for pod openshift-network-diagnostics/network-check-target-nf86x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:16.924189 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.923764 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr podName:9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d nodeName:}" failed. No retries permitted until 2026-07-21 12:11:18.923744433 +0000 UTC m=+5.093095705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhzhr" (UniqueName: "kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr") pod "network-check-target-nf86x" (UID: "9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:16.924189 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.924165 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:16.924690 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:16.924234 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret podName:426a4044-78b3-421b-abd3-85f56961434a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:18.924215238 +0000 UTC m=+5.093566498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret") pod "global-pull-secret-syncer-xmpjs" (UID: "426a4044-78b3-421b-abd3-85f56961434a") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:17.251781 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:17.251735 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 12:06:15 +0000 UTC" deadline="2028-03-13 21:31:01.126259753 +0000 UTC" Jul 21 12:11:17.251781 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:17.251780 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14433h19m43.874484911s" Jul 21 12:11:17.352125 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:17.352086 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:17.352336 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:17.352236 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:18.355058 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:18.355004 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:18.355569 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.355183 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:18.355569 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:18.355259 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:18.355569 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.355342 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:18.843534 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:18.842820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:18.843534 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.843024 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:18.843534 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.843099 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs podName:c5f44d8c-0fac-49cf-8e58-e8e33a638319 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:22.843077797 +0000 UTC m=+9.012429071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs") pod "network-metrics-daemon-jqth5" (UID: "c5f44d8c-0fac-49cf-8e58-e8e33a638319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:18.943831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:18.943924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.944084 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.944156 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret podName:426a4044-78b3-421b-abd3-85f56961434a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:22.944136718 +0000 UTC m=+9.113487977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret") pod "global-pull-secret-syncer-xmpjs" (UID: "426a4044-78b3-421b-abd3-85f56961434a") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.944592 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.944612 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.944625 2559 projected.go:196] Error preparing data for projected volume kube-api-access-fhzhr for pod openshift-network-diagnostics/network-check-target-nf86x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:18.944736 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:18.944697 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr podName:9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d nodeName:}" failed. No retries permitted until 2026-07-21 12:11:22.944680216 +0000 UTC m=+9.114031490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhzhr" (UniqueName: "kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr") pod "network-check-target-nf86x" (UID: "9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:19.352014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:19.351979 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:19.352206 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:19.352123 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:20.353952 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:20.353363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:20.353952 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:20.353416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:20.353952 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:20.353535 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:20.353952 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:20.353668 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:21.352781 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:21.352746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:21.352961 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:21.352891 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:22.357050 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:22.356936 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:22.357050 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:22.356936 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:22.357664 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.357113 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:22.357664 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.357217 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:22.877916 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:22.877871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:22.878125 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.878048 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:22.878202 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.878127 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs podName:c5f44d8c-0fac-49cf-8e58-e8e33a638319 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:30.878105685 +0000 UTC m=+17.047456955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs") pod "network-metrics-daemon-jqth5" (UID: "c5f44d8c-0fac-49cf-8e58-e8e33a638319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:22.978472 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:22.978427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:22.978695 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:22.978539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:22.978695 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.978676 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:22.978814 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.978736 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret podName:426a4044-78b3-421b-abd3-85f56961434a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:30.978721907 +0000 UTC m=+17.148073163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret") pod "global-pull-secret-syncer-xmpjs" (UID: "426a4044-78b3-421b-abd3-85f56961434a") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:22.979212 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.979083 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:11:22.979212 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.979113 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:11:22.979212 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.979128 2559 projected.go:196] Error preparing data for projected volume kube-api-access-fhzhr for pod openshift-network-diagnostics/network-check-target-nf86x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:22.979212 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:22.979190 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr podName:9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d nodeName:}" failed. No retries permitted until 2026-07-21 12:11:30.979172485 +0000 UTC m=+17.148523745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhzhr" (UniqueName: "kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr") pod "network-check-target-nf86x" (UID: "9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:23.352330 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:23.352234 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:23.352509 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:23.352406 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:24.354614 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:24.354577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:24.355768 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:24.355742 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:24.355919 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:24.355886 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:24.356055 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:24.356031 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:25.352052 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:25.352007 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:25.352313 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:25.352149 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:26.352998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:26.352962 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:26.353602 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:26.353100 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:26.353602 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:26.352962 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:26.353602 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:26.353189 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:27.352939 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:27.352857 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:27.353098 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:27.352992 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:28.352184 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:28.352146 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:28.352184 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:28.352184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:28.352435 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:28.352300 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:28.352485 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:28.352456 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:29.352239 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:29.352196 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:29.352726 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:29.352346 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:30.352507 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:30.352468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:30.352507 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:30.352504 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:30.353068 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:30.352624 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:30.353068 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:30.352750 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:30.938399 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:30.938356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:30.938602 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:30.938516 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:30.938602 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:30.938590 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs podName:c5f44d8c-0fac-49cf-8e58-e8e33a638319 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:46.93856875 +0000 UTC m=+33.107920029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs") pod "network-metrics-daemon-jqth5" (UID: "c5f44d8c-0fac-49cf-8e58-e8e33a638319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:31.039780 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:31.039740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:31.039978 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:31.039962 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:11:31.040062 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:31.039989 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:11:31.040062 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:31.040005 2559 projected.go:196] Error preparing data for projected volume kube-api-access-fhzhr for pod openshift-network-diagnostics/network-check-target-nf86x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:31.040175 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:31.040068 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr podName:9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d nodeName:}" failed. No retries permitted until 2026-07-21 12:11:47.040049097 +0000 UTC m=+33.209400354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhzhr" (UniqueName: "kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr") pod "network-check-target-nf86x" (UID: "9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:31.040175 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:31.040115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:31.040250 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:31.040214 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:31.040280 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:31.040269 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret podName:426a4044-78b3-421b-abd3-85f56961434a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:47.040255146 +0000 UTC m=+33.209606400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret") pod "global-pull-secret-syncer-xmpjs" (UID: "426a4044-78b3-421b-abd3-85f56961434a") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:31.352836 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:31.352760 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:31.353231 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:31.352868 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:32.352125 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:32.352092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:32.352309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:32.352092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:32.352309 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:32.352219 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:32.352309 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:32.352276 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:33.352834 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:33.352799 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:33.353351 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:33.352915 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:34.354045 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:34.353716 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:34.354510 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:34.353774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:34.354510 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:34.354105 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:34.354510 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:34.354185 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:35.352934 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:35.352894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:35.353124 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:35.353039 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:36.352515 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:36.352480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:36.353035 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:36.352615 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:36.353035 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:36.352669 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:36.353035 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:36.352803 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:37.353022 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.352804 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:37.353672 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:37.353100 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:37.519653 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.519539 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" event={"ID":"e306b331cd25c79c60bbf3257cbc2b52","Type":"ContainerStarted","Data":"72fe2a10d8a35f66d37c2b9773578d491d1cc85fd5a3455f5808ad72e84ca473"} Jul 21 12:11:37.521086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.521059 2559 generic.go:356] "Generic (PLEG): container finished" podID="08891ba6-492f-4dfd-a54c-b826dcf02d92" containerID="729cd60f77ac22d94d2a8d28d9b6c73a3edc3b502b235f470e7012e49370eddb" exitCode=0 Jul 21 12:11:37.521221 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.521149 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerDied","Data":"729cd60f77ac22d94d2a8d28d9b6c73a3edc3b502b235f470e7012e49370eddb"} Jul 21 12:11:37.522619 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.522590 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmb77" event={"ID":"2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d","Type":"ContainerStarted","Data":"4dcaea23054ff28cf25a24d0f81f02782e8650e95b90336ca8946ce659e1b4c2"} Jul 21 12:11:37.524179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.524083 2559 generic.go:356] "Generic (PLEG): container finished" podID="6fa6df74d0193c0424d08a53679b0d40" containerID="03e2ca53c3bf79184f18c3fa7c9a07730c811b6f5ce78f08e6ad63f47644b84d" exitCode=0 Jul 21 12:11:37.524179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.524113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" event={"ID":"6fa6df74d0193c0424d08a53679b0d40","Type":"ContainerDied","Data":"03e2ca53c3bf79184f18c3fa7c9a07730c811b6f5ce78f08e6ad63f47644b84d"} Jul 21 12:11:37.525792 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.525768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zpmfn" event={"ID":"14cd5476-2b2c-4e0b-bf29-78901a796492","Type":"ContainerStarted","Data":"ab0f915faadd84c19cb06d0244ba703ca895b46b8212de8ab1d688fb99ae4367"} Jul 21 12:11:37.527233 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.527210 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j5tm5" event={"ID":"206118a1-01d3-44ca-894f-0c229a671f20","Type":"ContainerStarted","Data":"e0dc0c29c6fa69607875b1bf934bbe71887498eebd8641879591a6751560a604"} Jul 21 12:11:37.528517 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.528493 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-94lz5" event={"ID":"c23a5fec-2f36-44cd-811f-2e5ba9534759","Type":"ContainerStarted","Data":"dd0f5c56d37d5b1bcdd7d560bee3a0c5ccf49198d384527fd0f307e4d275b3fb"} Jul 21 12:11:37.529695 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.529679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" event={"ID":"9939f893-b6d8-4c95-aef4-dbe6866eaabe","Type":"ContainerStarted","Data":"23c0e82e3d237b00f6825a801288adc9a20d8e2c3842709a1bdd8a006f4d063c"} Jul 21 12:11:37.530927 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.530903 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jkd6" event={"ID":"eae41061-6724-4683-b038-5495c849fa21","Type":"ContainerStarted","Data":"86c1ea12f31e4fe1ad746784d8ff433633367a754415502b1d12c330e381dd43"} Jul 21 12:11:37.534221 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534204 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:11:37.534535 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534513 2559 generic.go:356] "Generic (PLEG): container finished" podID="eef530bc-aba7-4b53-93d6-c8818dcffe5a" containerID="b5b75b79e167f87cc56e83ff81861fdca4a648d9d52f3f0a32ece62b3de9c687" exitCode=1 Jul 21 12:11:37.534626 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"8d722aca2e39517d26b37b39e7d920bc27d525c02146c075eac6ce5c868f505a"} Jul 21 12:11:37.534626 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534612 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"5728e5cf9501fe29dd20dc3f7d77ff16f52ba930e5afd782c153e44dfa06b717"} Jul 21 12:11:37.534708 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"a17575822ab209e7eea6dddf1890eb51f3679ae484cf6c8acde73109c3299c79"} Jul 21 12:11:37.534708 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"3489186f6fbfd8a198b537a4eba39ae9a83ee627e0675f5e85ee6b1aff20b7ec"} Jul 21 12:11:37.534708 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerDied","Data":"b5b75b79e167f87cc56e83ff81861fdca4a648d9d52f3f0a32ece62b3de9c687"} Jul 21 12:11:37.534800 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.534735 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"6aee39b4f2139c4d42ff84549be51d23f95c3f2fae38b97af69a650fcb721e8a"} Jul 21 12:11:37.550228 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.550166 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-97.ec2.internal" podStartSLOduration=22.550150072 podStartE2EDuration="22.550150072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:37.534989569 +0000 UTC m=+23.704340845" watchObservedRunningTime="2026-07-21 12:11:37.550150072 +0000 UTC m=+23.719501479" Jul 21 12:11:37.550438 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.550411 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j5tm5" podStartSLOduration=1.7764660060000002 podStartE2EDuration="22.550401661s" totalImagesPullingTime="20.773935655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:37.550120145 +0000 UTC m=+23.719471420" watchObservedRunningTime="2026-07-21 12:11:37.550401661 +0000 UTC m=+23.719752936" Jul 21 12:11:37.569595 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.569550 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zpmfn" podStartSLOduration=1.788681414 podStartE2EDuration="22.569534355s" totalImagesPullingTime="20.780852941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:37.569395559 +0000 UTC m=+23.738746836" watchObservedRunningTime="2026-07-21 12:11:37.569534355 +0000 UTC m=+23.738885640" Jul 21 12:11:37.586863 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.586802 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9jkd6" podStartSLOduration=1.781059892 podStartE2EDuration="22.58678157s" totalImagesPullingTime="20.805721678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:37.585943089 +0000 UTC m=+23.755294366" watchObservedRunningTime="2026-07-21 12:11:37.58678157 +0000 UTC m=+23.756132846" Jul 21 12:11:37.604369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.604098 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bmb77" podStartSLOduration=1.825197444 podStartE2EDuration="22.604078426s" totalImagesPullingTime="20.778880982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:37.603049739 +0000 UTC m=+23.772401017" watchObservedRunningTime="2026-07-21 12:11:37.604078426 +0000 UTC m=+23.773429703" Jul 21 12:11:37.673795 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:37.673729 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-94lz5" podStartSLOduration=1.886698992 podStartE2EDuration="22.67370527s" totalImagesPullingTime="20.787006278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:37.673311636 +0000 UTC m=+23.842662923" watchObservedRunningTime="2026-07-21 12:11:37.67370527 +0000 UTC m=+23.843056547" Jul 21 12:11:38.352435 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.352404 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:38.352611 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.352477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:38.352611 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:38.352580 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:38.352843 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:38.352689 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:38.437263 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.437232 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 12:11:38.539384 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.539292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" event={"ID":"9939f893-b6d8-4c95-aef4-dbe6866eaabe","Type":"ContainerStarted","Data":"dfd19aafa80f1e5cf31307812b6d42e6e28a90959c35b9f94b71cc0ff76b3fae"} Jul 21 12:11:38.540836 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.540804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-72rzb" event={"ID":"83a7f9b9-ee96-4874-bd0d-fda250113e12","Type":"ContainerStarted","Data":"7d12ebd0acc429de6754f91da4f3c6b6739d121b7d942026d1d6e3b3a5b8e57b"} Jul 21 12:11:38.542614 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.542535 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" event={"ID":"6fa6df74d0193c0424d08a53679b0d40","Type":"ContainerStarted","Data":"cc98e9a3157aecd90372ad9e0fe73e68bd4957b1e17016adc9e33c5ccde46bc8"} Jul 21 12:11:38.555284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.555235 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-72rzb" podStartSLOduration=2.732446197 podStartE2EDuration="23.555215497s" totalImagesPullingTime="20.8227693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:38.554742156 +0000 UTC m=+24.724093432" watchObservedRunningTime="2026-07-21 12:11:38.555215497 +0000 UTC m=+24.724566773" Jul 21 12:11:38.568708 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:38.568659 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-97.ec2.internal" podStartSLOduration=23.56864353 podStartE2EDuration="23.56864353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:38.568368711 +0000 UTC m=+24.737719986" watchObservedRunningTime="2026-07-21 12:11:38.56864353 +0000 UTC m=+24.737994805" Jul 21 12:11:39.276786 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.276605 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:39.277328 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.277304 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:39.277569 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.277459 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T12:11:38.437256401Z","UUID":"13110893-fd37-4e70-b5f3-aa7acb84886a","Handler":null,"Name":"","Endpoint":""} Jul 21 12:11:39.279842 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.279813 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 12:11:39.279987 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.279851 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 12:11:39.352246 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.352214 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:39.352421 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:39.352344 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:39.548502 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.548421 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:11:39.549149 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.549109 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"2e538ccd1518e5990ee3749a3309dbafd3cf73c2315e74e19c39356ad32c99cc"} Jul 21 12:11:39.549378 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.549351 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:39.549913 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:39.549896 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9jkd6" Jul 21 12:11:40.352273 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:40.352241 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:40.352505 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:40.352244 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:40.352505 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:40.352420 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:40.352606 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:40.352569 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:40.553623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:40.553583 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" event={"ID":"9939f893-b6d8-4c95-aef4-dbe6866eaabe","Type":"ContainerStarted","Data":"6f4e956cba90261b2cc451eb0edb07518608a8368dae26eefd0c97d9946a2cd6"} Jul 21 12:11:40.571227 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:40.571165 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj9fl" podStartSLOduration=7.415242016 podStartE2EDuration="25.571148598s" totalImagesPullingTime="18.155906582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:40.570407483 +0000 UTC m=+26.739758759" watchObservedRunningTime="2026-07-21 12:11:40.571148598 +0000 UTC m=+26.740499888" Jul 21 12:11:41.352812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:41.352774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:41.353010 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:41.352905 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:42.352899 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.352726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:42.353525 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.352782 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:42.353525 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:42.353008 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:42.353525 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:42.353084 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:42.561803 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.561731 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:11:42.562157 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.562131 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"3db9bbd07524ed14cbd7c060983ad21cafafdac742bc8742678545a919c0a5ae"} Jul 21 12:11:42.562436 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.562409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:42.562559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.562484 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:42.562559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.562529 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:42.562679 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.562656 2559 scope.go:117] "RemoveContainer" containerID="b5b75b79e167f87cc56e83ff81861fdca4a648d9d52f3f0a32ece62b3de9c687" Jul 21 12:11:42.564208 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.564182 2559 generic.go:356] "Generic (PLEG): container finished" podID="08891ba6-492f-4dfd-a54c-b826dcf02d92" containerID="1c478928b52ef74e78bf7f6ef44aa0ffca779d87c5c707c72ce196034f546c02" exitCode=0 Jul 21 12:11:42.564314 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.564228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerDied","Data":"1c478928b52ef74e78bf7f6ef44aa0ffca779d87c5c707c72ce196034f546c02"} Jul 21 12:11:42.579676 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.579648 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:42.579874 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:42.579859 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:11:43.352690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:43.352653 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:43.352929 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:43.352807 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:43.569964 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:43.569918 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:11:43.570442 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:43.570418 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" event={"ID":"eef530bc-aba7-4b53-93d6-c8818dcffe5a","Type":"ContainerStarted","Data":"c510df90d7ab9085fefb3c9b9d01183121057b99697ab6b95f85df25adb3eda5"} Jul 21 12:11:43.603246 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:43.603143 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" podStartSLOduration=7.701637956 podStartE2EDuration="28.603118895s" totalImagesPullingTime="20.901480939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:43.601559456 +0000 UTC m=+29.770910725" watchObservedRunningTime="2026-07-21 12:11:43.603118895 +0000 UTC m=+29.772470175" Jul 21 12:11:44.214338 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.214134 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xmpjs"] Jul 21 12:11:44.214534 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.214480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:44.214596 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:44.214574 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:44.217216 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.217189 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nf86x"] Jul 21 12:11:44.217378 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.217311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:44.217459 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:44.217433 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:44.217999 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.217975 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jqth5"] Jul 21 12:11:44.218096 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.218081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:44.218189 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:44.218166 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:44.574682 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.574654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerStarted","Data":"820bc4c2a4a981a79a4b0b428dd2006e18697650cecccf575f23b2d436197784"} Jul 21 12:11:44.647354 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.647322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:44.752237 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:44.752156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:45.352707 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:45.352674 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:45.352884 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:45.352781 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:45.578640 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:45.578599 2559 generic.go:356] "Generic (PLEG): container finished" podID="08891ba6-492f-4dfd-a54c-b826dcf02d92" containerID="820bc4c2a4a981a79a4b0b428dd2006e18697650cecccf575f23b2d436197784" exitCode=0 Jul 21 12:11:45.579061 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:45.578674 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerDied","Data":"820bc4c2a4a981a79a4b0b428dd2006e18697650cecccf575f23b2d436197784"} Jul 21 12:11:46.352564 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:46.352483 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:46.352711 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:46.352593 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:46.352711 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:46.352654 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:46.352803 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:46.352779 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:46.960349 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:46.960268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:46.960679 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:46.960415 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:46.960679 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:46.960491 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs podName:c5f44d8c-0fac-49cf-8e58-e8e33a638319 nodeName:}" failed. No retries permitted until 2026-07-21 12:12:18.960467722 +0000 UTC m=+65.129818990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs") pod "network-metrics-daemon-jqth5" (UID: "c5f44d8c-0fac-49cf-8e58-e8e33a638319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 12:11:47.061224 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:47.061184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:47.061404 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:47.061237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:47.061404 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:47.061336 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:47.061404 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:47.061400 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret podName:426a4044-78b3-421b-abd3-85f56961434a nodeName:}" failed. No retries permitted until 2026-07-21 12:12:19.061385681 +0000 UTC m=+65.230736935 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret") pod "global-pull-secret-syncer-xmpjs" (UID: "426a4044-78b3-421b-abd3-85f56961434a") : object "kube-system"/"original-pull-secret" not registered Jul 21 12:11:47.061529 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:47.061348 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 12:11:47.061529 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:47.061436 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 12:11:47.061529 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:47.061445 2559 projected.go:196] Error preparing data for projected volume kube-api-access-fhzhr for pod openshift-network-diagnostics/network-check-target-nf86x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:47.061529 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:47.061486 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr podName:9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d nodeName:}" failed. No retries permitted until 2026-07-21 12:12:19.06147526 +0000 UTC m=+65.230826527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fhzhr" (UniqueName: "kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr") pod "network-check-target-nf86x" (UID: "9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 12:11:47.091081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:47.091047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:47.158535 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:47.158498 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:47.352610 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:47.352518 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:47.352788 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:47.352660 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xmpjs" podUID="426a4044-78b3-421b-abd3-85f56961434a" Jul 21 12:11:47.586090 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:47.586055 2559 generic.go:356] "Generic (PLEG): container finished" podID="08891ba6-492f-4dfd-a54c-b826dcf02d92" containerID="22f86f1f2ff41ee76bdf9fe464030216c342817c2f2e3dec41113031d86df838" exitCode=0 Jul 21 12:11:47.586254 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:47.586101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerDied","Data":"22f86f1f2ff41ee76bdf9fe464030216c342817c2f2e3dec41113031d86df838"} Jul 21 12:11:48.353009 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:48.352972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:48.353460 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:48.353080 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf86x" podUID="9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d" Jul 21 12:11:48.353460 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:48.353128 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:48.353460 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:48.353229 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jqth5" podUID="c5f44d8c-0fac-49cf-8e58-e8e33a638319" Jul 21 12:11:49.124004 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.123974 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-97.ec2.internal" event="NodeReady" Jul 21 12:11:49.124179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.124110 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 12:11:49.172229 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.172191 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f95845fff-stkqs"] Jul 21 12:11:49.176469 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.176441 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.179087 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.179056 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 12:11:49.179210 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.179147 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l94b2\"" Jul 21 12:11:49.179210 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.179204 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 12:11:49.179322 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.179277 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 12:11:49.181532 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.181494 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nbhbz"] Jul 21 12:11:49.183748 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.183581 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h4jt2"] Jul 21 12:11:49.183748 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.183710 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.185219 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.185197 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 12:11:49.185749 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.185731 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.186479 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.186316 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 12:11:49.186479 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.186384 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 12:11:49.187163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.187146 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fnnf7\"" Jul 21 12:11:49.188231 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.188211 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 12:11:49.188389 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.188354 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 12:11:49.188874 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.188855 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6rff6\"" Jul 21 12:11:49.189185 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.189166 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 12:11:49.190044 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.189992 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f95845fff-stkqs"] Jul 21 12:11:49.195233 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.195208 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4jt2"] Jul 21 12:11:49.198664 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.198580 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nbhbz"] Jul 21 12:11:49.277081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.276991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-trusted-ca\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277030 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/680cbf07-e123-4188-8052-8d691df7d859-cert\") pod \"ingress-canary-h4jt2\" (UID: \"680cbf07-e123-4188-8052-8d691df7d859\") " pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.277081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-installation-pull-secrets\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277351 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzv7\" (UniqueName: \"kubernetes.io/projected/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-kube-api-access-nmzv7\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.277351 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277351 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277225 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-certificates\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277351 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdttm\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-kube-api-access-rdttm\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277351 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277317 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzdz7\" (UniqueName: \"kubernetes.io/projected/680cbf07-e123-4188-8052-8d691df7d859-kube-api-access-xzdz7\") pod \"ingress-canary-h4jt2\" (UID: \"680cbf07-e123-4188-8052-8d691df7d859\") " pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.277546 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277351 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-tmp-dir\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.277546 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277384 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-bound-sa-token\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277546 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277513 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-image-registry-private-configuration\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277546 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.277714 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277563 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d053645-ecd5-4ca5-a8b2-569179b19b6a-ca-trust-extracted\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.277714 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.277584 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-config-volume\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.351973 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.351914 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:11:49.355269 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.355249 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 12:11:49.378451 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-installation-pull-secrets\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.378637 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzv7\" (UniqueName: \"kubernetes.io/projected/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-kube-api-access-nmzv7\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.378637 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.378637 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-certificates\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.378637 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdttm\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-kube-api-access-rdttm\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.378637 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzdz7\" (UniqueName: \"kubernetes.io/projected/680cbf07-e123-4188-8052-8d691df7d859-kube-api-access-xzdz7\") pod \"ingress-canary-h4jt2\" (UID: \"680cbf07-e123-4188-8052-8d691df7d859\") " pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.378637 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-tmp-dir\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.378902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-bound-sa-token\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.378902 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.378678 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:11:49.378902 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.378695 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f95845fff-stkqs: secret "image-registry-tls" not found Jul 21 12:11:49.378902 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.378752 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls podName:1d053645-ecd5-4ca5-a8b2-569179b19b6a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:49.878733579 +0000 UTC m=+36.048084855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls") pod "image-registry-5f95845fff-stkqs" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a") : secret "image-registry-tls" not found Jul 21 12:11:49.378902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378772 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-image-registry-private-configuration\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.378902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.378902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.378836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d053645-ecd5-4ca5-a8b2-569179b19b6a-ca-trust-extracted\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.379578 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.379544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-tmp-dir\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.379968 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.379702 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d053645-ecd5-4ca5-a8b2-569179b19b6a-ca-trust-extracted\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.380073 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.380013 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:11:49.380121 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.380080 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls podName:a413545d-fdc9-45c1-b1b9-79cb9965cfa7 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:49.880060424 +0000 UTC m=+36.049411695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls") pod "dns-default-nbhbz" (UID: "a413545d-fdc9-45c1-b1b9-79cb9965cfa7") : secret "dns-default-metrics-tls" not found Jul 21 12:11:49.380484 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.380374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-config-volume\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.380484 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.380432 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-trusted-ca\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.380484 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.380459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/680cbf07-e123-4188-8052-8d691df7d859-cert\") pod \"ingress-canary-h4jt2\" (UID: \"680cbf07-e123-4188-8052-8d691df7d859\") " pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.380484 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.380461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-certificates\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.381229 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.381200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-config-volume\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.382409 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.382383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-trusted-ca\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.383843 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.383813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-image-registry-private-configuration\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.385702 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.385665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-installation-pull-secrets\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.385824 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.385800 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/680cbf07-e123-4188-8052-8d691df7d859-cert\") pod \"ingress-canary-h4jt2\" (UID: \"680cbf07-e123-4188-8052-8d691df7d859\") " pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.387963 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.387932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzv7\" (UniqueName: \"kubernetes.io/projected/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-kube-api-access-nmzv7\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.391177 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.391154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-bound-sa-token\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.391496 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.391477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzdz7\" (UniqueName: \"kubernetes.io/projected/680cbf07-e123-4188-8052-8d691df7d859-kube-api-access-xzdz7\") pod \"ingress-canary-h4jt2\" (UID: \"680cbf07-e123-4188-8052-8d691df7d859\") " pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.391574 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.391514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdttm\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-kube-api-access-rdttm\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.506737 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.506694 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4jt2" Jul 21 12:11:49.520651 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.520618 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:49.665282 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.665249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4jt2"] Jul 21 12:11:49.669788 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:11:49.669759 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680cbf07_e123_4188_8052_8d691df7d859.slice/crio-c661cbdd51615b444f96a848aefee170d1b8f4b9020236455b1a3079b5c83cba WatchSource:0}: Error finding container c661cbdd51615b444f96a848aefee170d1b8f4b9020236455b1a3079b5c83cba: Status 404 returned error can't find the container with id c661cbdd51615b444f96a848aefee170d1b8f4b9020236455b1a3079b5c83cba Jul 21 12:11:49.884654 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.884560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:49.884654 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:49.884625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:49.884862 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.884728 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:11:49.884862 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.884757 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:11:49.884862 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.884771 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f95845fff-stkqs: secret "image-registry-tls" not found Jul 21 12:11:49.884862 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.884810 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls podName:a413545d-fdc9-45c1-b1b9-79cb9965cfa7 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:50.884787983 +0000 UTC m=+37.054139250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls") pod "dns-default-nbhbz" (UID: "a413545d-fdc9-45c1-b1b9-79cb9965cfa7") : secret "dns-default-metrics-tls" not found Jul 21 12:11:49.884862 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:49.884831 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls podName:1d053645-ecd5-4ca5-a8b2-569179b19b6a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:50.884821318 +0000 UTC m=+37.054172577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls") pod "image-registry-5f95845fff-stkqs" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a") : secret "image-registry-tls" not found Jul 21 12:11:50.356248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.356106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:11:50.356248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.356139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:11:50.358793 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.358770 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 12:11:50.358793 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.358788 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2w9zn\"" Jul 21 12:11:50.359281 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.359258 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 12:11:50.359376 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.359320 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8v447\"" Jul 21 12:11:50.359376 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.359349 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 12:11:50.377189 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.377158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:50.595910 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.595875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4jt2" event={"ID":"680cbf07-e123-4188-8052-8d691df7d859","Type":"ContainerStarted","Data":"c661cbdd51615b444f96a848aefee170d1b8f4b9020236455b1a3079b5c83cba"} Jul 21 12:11:50.891738 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.891698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:50.891967 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:50.891798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:50.891967 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:50.891885 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:11:50.891967 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:50.891911 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f95845fff-stkqs: secret "image-registry-tls" not found Jul 21 12:11:50.891967 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:50.891929 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:11:50.892175 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:50.891997 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls podName:1d053645-ecd5-4ca5-a8b2-569179b19b6a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:52.891975121 +0000 UTC m=+39.061326382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls") pod "image-registry-5f95845fff-stkqs" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a") : secret "image-registry-tls" not found Jul 21 12:11:50.892175 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:50.892017 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls podName:a413545d-fdc9-45c1-b1b9-79cb9965cfa7 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:52.892006484 +0000 UTC m=+39.061357743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls") pod "dns-default-nbhbz" (UID: "a413545d-fdc9-45c1-b1b9-79cb9965cfa7") : secret "dns-default-metrics-tls" not found Jul 21 12:11:52.906468 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:52.906429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:52.907011 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:52.906531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:52.907011 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:52.906606 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:11:52.907011 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:52.906657 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:11:52.907011 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:52.906671 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f95845fff-stkqs: secret "image-registry-tls" not found Jul 21 12:11:52.907011 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:52.906694 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls podName:a413545d-fdc9-45c1-b1b9-79cb9965cfa7 nodeName:}" failed. No retries permitted until 2026-07-21 12:11:56.906671261 +0000 UTC m=+43.076022532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls") pod "dns-default-nbhbz" (UID: "a413545d-fdc9-45c1-b1b9-79cb9965cfa7") : secret "dns-default-metrics-tls" not found Jul 21 12:11:52.907011 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:52.906718 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls podName:1d053645-ecd5-4ca5-a8b2-569179b19b6a nodeName:}" failed. No retries permitted until 2026-07-21 12:11:56.906705619 +0000 UTC m=+43.076056882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls") pod "image-registry-5f95845fff-stkqs" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a") : secret "image-registry-tls" not found Jul 21 12:11:53.475309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:53.475199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:53.982290 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:53.982072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:54.078395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.077714 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:54.607404 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.607370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4jt2" event={"ID":"680cbf07-e123-4188-8052-8d691df7d859","Type":"ContainerStarted","Data":"e04e420a5b861ca4e1938a44da73124cc618b17cdaa8fc906261b4c9cee9eaa8"} Jul 21 12:11:54.610100 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.610057 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerDied","Data":"cddf8659650aec52e470bac153e2674912865a0f5e9707d81a207fa3cf044dfe"} Jul 21 12:11:54.610259 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.610037 2559 generic.go:356] "Generic (PLEG): container finished" podID="08891ba6-492f-4dfd-a54c-b826dcf02d92" containerID="cddf8659650aec52e470bac153e2674912865a0f5e9707d81a207fa3cf044dfe" exitCode=0 Jul 21 12:11:54.643239 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.643192 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h4jt2" podStartSLOduration=1.412937741 podStartE2EDuration="5.643172167s" totalImagesPullingTime="4.230234426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:54.642670934 +0000 UTC m=+40.812022212" watchObservedRunningTime="2026-07-21 12:11:54.643172167 +0000 UTC m=+40.812523434" Jul 21 12:11:54.814003 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.813971 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kxw7m"] Jul 21 12:11:54.825353 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.825327 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:54.826929 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.826898 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kxw7m"] Jul 21 12:11:54.827384 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.827344 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 12:11:54.827595 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.827578 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 12:11:54.827595 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.827586 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 12:11:54.827741 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.827617 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2gbfg\"" Jul 21 12:11:54.827741 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.827640 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 12:11:54.844149 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.844123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:54.900085 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.900054 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:54.924886 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.924858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-crio-socket\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:54.925061 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.924891 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:54.925061 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.924914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-data-volume\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:54.925061 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.924930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvt2\" (UniqueName: \"kubernetes.io/projected/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-api-access-vzvt2\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:54.925194 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:54.925074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026194 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.026151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-crio-socket\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026194 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.026196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.026217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-data-volume\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.026240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvt2\" (UniqueName: \"kubernetes.io/projected/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-api-access-vzvt2\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.026311 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.026390 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:11:55.526370853 +0000 UTC m=+41.695722121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : configmap "kube-rbac-proxy" not found Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.026430 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.026434 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-crio-socket\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.026547 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.026568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-data-volume\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.026690 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.026589 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:11:55.526576457 +0000 UTC m=+41.695927711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : secret "insights-runtime-extractor-tls" not found Jul 21 12:11:55.036727 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.036690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvt2\" (UniqueName: \"kubernetes.io/projected/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-api-access-vzvt2\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.141343 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.141246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.224671 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.224640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.325657 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.325621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.387877 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.387847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.505700 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.505666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.532181 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.532144 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.532321 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.532202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:55.532321 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.532263 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 12:11:55.532321 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.532310 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:11:55.532440 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.532324 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:11:56.532309139 +0000 UTC m=+42.701660393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : configmap "kube-rbac-proxy" not found Jul 21 12:11:55.532440 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:55.532357 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:11:56.5323454 +0000 UTC m=+42.701696655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : secret "insights-runtime-extractor-tls" not found Jul 21 12:11:55.598584 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.598554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.615295 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.615266 2559 generic.go:356] "Generic (PLEG): container finished" podID="08891ba6-492f-4dfd-a54c-b826dcf02d92" containerID="e30325ed337c9cbd9c75c3d6d8f8997677e26041f2249e76b4a0e618d3069fa4" exitCode=0 Jul 21 12:11:55.615455 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.615347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerDied","Data":"e30325ed337c9cbd9c75c3d6d8f8997677e26041f2249e76b4a0e618d3069fa4"} Jul 21 12:11:55.751904 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.751801 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.852669 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.852589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:55.958967 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:55.958917 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:56.059998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.059943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:56.536157 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.536126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:56.542229 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.542208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:56.542303 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.542279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:56.542350 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.542338 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 12:11:56.542399 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.542387 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:11:56.542433 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.542415 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:11:58.54239509 +0000 UTC m=+44.711746348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : configmap "kube-rbac-proxy" not found Jul 21 12:11:56.542473 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.542434 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:11:58.542424639 +0000 UTC m=+44.711775910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : secret "insights-runtime-extractor-tls" not found Jul 21 12:11:56.621408 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.621373 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74znp" event={"ID":"08891ba6-492f-4dfd-a54c-b826dcf02d92","Type":"ContainerStarted","Data":"c196d7c0b296a65f9a5a448ac163e0d6bd0c53166a18812179b43a22f08fe8cd"} Jul 21 12:11:56.636977 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.636931 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:56.646422 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.646378 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-74znp" podStartSLOduration=7.112009509 podStartE2EDuration="41.646364297s" totalImagesPullingTime="34.534354788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:11:56.645481667 +0000 UTC m=+42.814832942" watchObservedRunningTime="2026-07-21 12:11:56.646364297 +0000 UTC m=+42.815715573" Jul 21 12:11:56.805166 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.805091 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j5tm5_206118a1-01d3-44ca-894f-0c229a671f20/dns-node-resolver/0.log" Jul 21 12:11:56.889741 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.889699 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:56.946735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.946702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:11:56.946903 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.946783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:11:56.946903 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.946867 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:11:56.946990 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.946924 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls podName:a413545d-fdc9-45c1-b1b9-79cb9965cfa7 nodeName:}" failed. No retries permitted until 2026-07-21 12:12:04.94690938 +0000 UTC m=+51.116260633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls") pod "dns-default-nbhbz" (UID: "a413545d-fdc9-45c1-b1b9-79cb9965cfa7") : secret "dns-default-metrics-tls" not found Jul 21 12:11:56.946990 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.946925 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:11:56.946990 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.946943 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f95845fff-stkqs: secret "image-registry-tls" not found Jul 21 12:11:56.947087 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:56.947013 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls podName:1d053645-ecd5-4ca5-a8b2-569179b19b6a nodeName:}" failed. No retries permitted until 2026-07-21 12:12:04.94699827 +0000 UTC m=+51.116349537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls") pod "image-registry-5f95845fff-stkqs" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a") : secret "image-registry-tls" not found Jul 21 12:11:56.991322 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:56.991285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:57.302025 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:57.301995 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:57.351575 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:57.351545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:57.594800 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:57.594714 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:57.605375 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:57.605356 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmb77_2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d/node-ca/0.log" Jul 21 12:11:57.693584 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:57.693550 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:58.452361 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:58.452322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:11:58.560864 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:58.560833 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:58.560995 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:58.560922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:11:58.561043 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:58.561021 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:11:58.561099 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:58.561089 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:12:02.56107268 +0000 UTC m=+48.730423934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : secret "insights-runtime-extractor-tls" not found Jul 21 12:11:58.561137 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:58.561023 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 12:11:58.561169 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:11:58.561155 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:12:02.561144188 +0000 UTC m=+48.730495443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : configmap "kube-rbac-proxy" not found Jul 21 12:11:58.605724 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:58.605699 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4jt2_680cbf07-e123-4188-8052-8d691df7d859/serve-healthcheck-canary/0.log" Jul 21 12:11:58.888086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:11:58.888053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:00.671167 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:00.671131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:02.404903 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.404870 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:02.592524 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.592487 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:12:02.592696 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.592550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:12:02.592696 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:02.592649 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 12:12:02.592780 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:02.592703 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls podName:e9cdf497-3071-44fc-8a5e-9dbf895ba5db nodeName:}" failed. No retries permitted until 2026-07-21 12:12:10.592689715 +0000 UTC m=+56.762040969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kxw7m" (UID: "e9cdf497-3071-44fc-8a5e-9dbf895ba5db") : secret "insights-runtime-extractor-tls" not found Jul 21 12:12:02.593078 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.593058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:12:02.634142 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.634109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:02.741598 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.741568 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:02.843932 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.843897 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:02.950165 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:02.950135 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:04.039498 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:04.039460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:04.119272 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:04.119241 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:04.287247 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:04.287217 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:04.492218 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:04.492191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:04.702503 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:04.702471 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:04.892613 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:04.892538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:05.013755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:05.013455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:12:05.013964 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:05.013810 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 12:12:05.013964 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:05.013829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") pod \"image-registry-5f95845fff-stkqs\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:12:05.013964 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:05.013866 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls podName:a413545d-fdc9-45c1-b1b9-79cb9965cfa7 nodeName:}" failed. No retries permitted until 2026-07-21 12:12:21.013851027 +0000 UTC m=+67.183202280 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls") pod "dns-default-nbhbz" (UID: "a413545d-fdc9-45c1-b1b9-79cb9965cfa7") : secret "dns-default-metrics-tls" not found Jul 21 12:12:05.013964 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:05.013925 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 12:12:05.013964 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:05.013936 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f95845fff-stkqs: secret "image-registry-tls" not found Jul 21 12:12:05.014161 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:05.013996 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls podName:1d053645-ecd5-4ca5-a8b2-569179b19b6a nodeName:}" failed. No retries permitted until 2026-07-21 12:12:21.013984425 +0000 UTC m=+67.183335679 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls") pod "image-registry-5f95845fff-stkqs" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a") : secret "image-registry-tls" not found Jul 21 12:12:05.090796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:05.090761 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:06.087315 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:06.087283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:06.140056 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:06.140026 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:06.290133 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:06.290100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:06.495165 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:06.495131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:06.685619 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:06.685578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:06.894258 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:06.894172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:06.951258 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:06.951227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:07.096080 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:07.096049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:07.890381 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:07.890351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:08.297929 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:08.297897 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:08.588315 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:08.588236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:08.691082 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:08.691050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:09.094158 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:09.094127 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:09.399181 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:09.399108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:09.487278 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:09.487249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:09.893686 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:09.893654 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:10.091242 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:10.091210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:10.656239 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:10.656212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:12:10.658421 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:10.658403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9cdf497-3071-44fc-8a5e-9dbf895ba5db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kxw7m\" (UID: \"e9cdf497-3071-44fc-8a5e-9dbf895ba5db\") " pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:12:10.762900 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:10.762864 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kxw7m" Jul 21 12:12:10.887835 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:10.887805 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kxw7m"] Jul 21 12:12:10.890881 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:10.890855 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cdf497_3071_44fc_8a5e_9dbf895ba5db.slice/crio-5d25cb4f2879a924a0645544ab5d5dbae8589a23d7237a32eb5a8ccae7d11332 WatchSource:0}: Error finding container 5d25cb4f2879a924a0645544ab5d5dbae8589a23d7237a32eb5a8ccae7d11332: Status 404 returned error can't find the container with id 5d25cb4f2879a924a0645544ab5d5dbae8589a23d7237a32eb5a8ccae7d11332 Jul 21 12:12:11.558663 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:11.558614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:11.613224 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:11.613191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:11.656977 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:11.656921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxw7m" event={"ID":"e9cdf497-3071-44fc-8a5e-9dbf895ba5db","Type":"ContainerStarted","Data":"5d25cb4f2879a924a0645544ab5d5dbae8589a23d7237a32eb5a8ccae7d11332"} Jul 21 12:12:11.938680 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:11.938605 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:12.939876 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:12.939824 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:13.036959 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:13.036912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:13.138156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:13.138121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:14.585833 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:14.585807 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qs5s" Jul 21 12:12:17.673786 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.673747 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxw7m" event={"ID":"e9cdf497-3071-44fc-8a5e-9dbf895ba5db","Type":"ContainerStarted","Data":"2b063a9ed9faaad5b78dbcb04f5f570015828ff73cfbd4a7ebe97a36437b1862"} Jul 21 12:12:17.786337 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.786036 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f95845fff-stkqs"] Jul 21 12:12:17.786337 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:17.786290 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5f95845fff-stkqs" podUID="1d053645-ecd5-4ca5-a8b2-569179b19b6a" Jul 21 12:12:17.801049 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.800857 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv"] Jul 21 12:12:17.803349 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.803294 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:17.805011 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.804984 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862"] Jul 21 12:12:17.806157 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.806091 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 12:12:17.806738 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.806608 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 12:12:17.806738 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.806698 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 12:12:17.806914 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.806761 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 12:12:17.807122 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.807101 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:17.809431 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.809407 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 21 12:12:17.809632 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.809618 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6rmxb\"" Jul 21 12:12:17.844722 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.838436 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:17.844722 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.838522 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv"] Jul 21 12:12:17.844722 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.839275 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862"] Jul 21 12:12:17.893862 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.893831 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-644f4d8c7f-t5c6h"] Jul 21 12:12:17.895682 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.895650 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.910272 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.910247 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644f4d8c7f-t5c6h"] Jul 21 12:12:17.914273 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b21f7106-a24b-4cfd-b6d0-9f505af3368e-registry-certificates\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.914428 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b21f7106-a24b-4cfd-b6d0-9f505af3368e-image-registry-private-configuration\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.914428 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbj94\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-kube-api-access-mbj94\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.914428 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b21f7106-a24b-4cfd-b6d0-9f505af3368e-trusted-ca\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.914562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914438 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-registry-tls\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.914562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e936553e-e7fe-415f-ae4a-d7a21b5bff69-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dc9dffdfb-z8862\" (UID: \"e936553e-e7fe-415f-ae4a-d7a21b5bff69\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:17.914562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b21f7106-a24b-4cfd-b6d0-9f505af3368e-installation-pull-secrets\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.914562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:17.914736 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5hs\" (UniqueName: \"kubernetes.io/projected/e936553e-e7fe-415f-ae4a-d7a21b5bff69-kube-api-access-tx5hs\") pod \"managed-serviceaccount-addon-agent-7dc9dffdfb-z8862\" (UID: \"e936553e-e7fe-415f-ae4a-d7a21b5bff69\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:17.914736 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdpj\" (UniqueName: \"kubernetes.io/projected/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-kube-api-access-zmdpj\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:17.914834 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-tmp\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:17.914897 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b21f7106-a24b-4cfd-b6d0-9f505af3368e-ca-trust-extracted\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.914897 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.914860 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-bound-sa-token\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:17.931385 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:17.931310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:18.016360 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdpj\" (UniqueName: \"kubernetes.io/projected/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-kube-api-access-zmdpj\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:18.016553 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-tmp\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:18.016553 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b21f7106-a24b-4cfd-b6d0-9f505af3368e-ca-trust-extracted\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016553 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-bound-sa-token\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b21f7106-a24b-4cfd-b6d0-9f505af3368e-registry-certificates\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016689 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b21f7106-a24b-4cfd-b6d0-9f505af3368e-image-registry-private-configuration\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbj94\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-kube-api-access-mbj94\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016758 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b21f7106-a24b-4cfd-b6d0-9f505af3368e-trusted-ca\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-registry-tls\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016832 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e936553e-e7fe-415f-ae4a-d7a21b5bff69-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dc9dffdfb-z8862\" (UID: \"e936553e-e7fe-415f-ae4a-d7a21b5bff69\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:18.016902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b21f7106-a24b-4cfd-b6d0-9f505af3368e-installation-pull-secrets\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.016902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:18.017173 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5hs\" (UniqueName: \"kubernetes.io/projected/e936553e-e7fe-415f-ae4a-d7a21b5bff69-kube-api-access-tx5hs\") pod \"managed-serviceaccount-addon-agent-7dc9dffdfb-z8862\" (UID: \"e936553e-e7fe-415f-ae4a-d7a21b5bff69\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:18.017173 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b21f7106-a24b-4cfd-b6d0-9f505af3368e-ca-trust-extracted\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.017173 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.016969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-tmp\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:18.017597 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.017536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b21f7106-a24b-4cfd-b6d0-9f505af3368e-registry-certificates\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.018251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.018208 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b21f7106-a24b-4cfd-b6d0-9f505af3368e-trusted-ca\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.020006 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.019937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b21f7106-a24b-4cfd-b6d0-9f505af3368e-image-registry-private-configuration\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.020187 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.020166 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e936553e-e7fe-415f-ae4a-d7a21b5bff69-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dc9dffdfb-z8862\" (UID: \"e936553e-e7fe-415f-ae4a-d7a21b5bff69\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:18.020257 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.020191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-registry-tls\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.020353 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.020329 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b21f7106-a24b-4cfd-b6d0-9f505af3368e-installation-pull-secrets\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.020595 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.020573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:18.026240 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.026211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-bound-sa-token\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.026404 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.026272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdpj\" (UniqueName: \"kubernetes.io/projected/2900ce51-ea8b-45fd-8f32-5f9ad5d563ef-kube-api-access-zmdpj\") pod \"klusterlet-addon-workmgr-6dd99bffc-k24jv\" (UID: \"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:18.026729 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.026698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5hs\" (UniqueName: \"kubernetes.io/projected/e936553e-e7fe-415f-ae4a-d7a21b5bff69-kube-api-access-tx5hs\") pod \"managed-serviceaccount-addon-agent-7dc9dffdfb-z8862\" (UID: \"e936553e-e7fe-415f-ae4a-d7a21b5bff69\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:18.027091 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.027072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbj94\" (UniqueName: \"kubernetes.io/projected/b21f7106-a24b-4cfd-b6d0-9f505af3368e-kube-api-access-mbj94\") pod \"image-registry-644f4d8c7f-t5c6h\" (UID: \"b21f7106-a24b-4cfd-b6d0-9f505af3368e\") " pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.031535 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.031510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:18.138002 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.137965 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:18.162176 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.162136 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" Jul 21 12:12:18.208255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.208030 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l94b2\"" Jul 21 12:12:18.216916 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.216371 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.288607 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.288488 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv"] Jul 21 12:12:18.320329 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.320267 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862"] Jul 21 12:12:18.325472 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:18.325447 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode936553e_e7fe_415f_ae4a_d7a21b5bff69.slice/crio-f36fc08e5f44c5e630a82ae6fab1da343660a88f4cf5c07ce1704c25872c83ce WatchSource:0}: Error finding container f36fc08e5f44c5e630a82ae6fab1da343660a88f4cf5c07ce1704c25872c83ce: Status 404 returned error can't find the container with id f36fc08e5f44c5e630a82ae6fab1da343660a88f4cf5c07ce1704c25872c83ce Jul 21 12:12:18.378865 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.378787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644f4d8c7f-t5c6h"] Jul 21 12:12:18.383580 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:18.383534 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21f7106_a24b_4cfd_b6d0_9f505af3368e.slice/crio-0dc1fac31e2a1a5de89ee2a0c45c7e10cdb351e249242502af99c5c10999dfff WatchSource:0}: Error finding container 0dc1fac31e2a1a5de89ee2a0c45c7e10cdb351e249242502af99c5c10999dfff: Status 404 returned error can't find the container with id 0dc1fac31e2a1a5de89ee2a0c45c7e10cdb351e249242502af99c5c10999dfff Jul 21 12:12:18.663056 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.663027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:18.679280 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.679242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" event={"ID":"b21f7106-a24b-4cfd-b6d0-9f505af3368e","Type":"ContainerStarted","Data":"ff6962d8a4f642faa6edd2aa6a3f50a84de2498ec30603ba60e2a8fd54a0cfe9"} Jul 21 12:12:18.679681 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.679306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" event={"ID":"b21f7106-a24b-4cfd-b6d0-9f505af3368e","Type":"ContainerStarted","Data":"0dc1fac31e2a1a5de89ee2a0c45c7e10cdb351e249242502af99c5c10999dfff"} Jul 21 12:12:18.679681 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.679332 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:18.680666 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.680634 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" event={"ID":"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef","Type":"ContainerStarted","Data":"8d6d36cab8fd0773520696e8fac7503c4eecb900488e5360fb45fba9e95004cc"} Jul 21 12:12:18.681789 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.681750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" event={"ID":"e936553e-e7fe-415f-ae4a-d7a21b5bff69","Type":"ContainerStarted","Data":"f36fc08e5f44c5e630a82ae6fab1da343660a88f4cf5c07ce1704c25872c83ce"} Jul 21 12:12:18.681900 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.681794 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:12:18.692092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.692062 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:12:18.705926 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.705850 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" podStartSLOduration=1.705828602 podStartE2EDuration="1.705828602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:18.704470855 +0000 UTC m=+64.873822146" watchObservedRunningTime="2026-07-21 12:12:18.705828602 +0000 UTC m=+64.875179859" Jul 21 12:12:18.721861 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.721835 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-installation-pull-secrets\") pod \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " Jul 21 12:12:18.722056 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.721879 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdttm\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-kube-api-access-rdttm\") pod \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " Jul 21 12:12:18.722056 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.721910 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d053645-ecd5-4ca5-a8b2-569179b19b6a-ca-trust-extracted\") pod \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " Jul 21 12:12:18.722221 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.722188 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d053645-ecd5-4ca5-a8b2-569179b19b6a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1d053645-ecd5-4ca5-a8b2-569179b19b6a" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:12:18.722335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.722321 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-bound-sa-token\") pod \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " Jul 21 12:12:18.722405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.722367 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-image-registry-private-configuration\") pod \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " Jul 21 12:12:18.722461 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.722408 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-trusted-ca\") pod \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " Jul 21 12:12:18.722509 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.722457 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-certificates\") pod \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\" (UID: \"1d053645-ecd5-4ca5-a8b2-569179b19b6a\") " Jul 21 12:12:18.722864 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.722835 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d053645-ecd5-4ca5-a8b2-569179b19b6a-ca-trust-extracted\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:18.723670 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.723623 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1d053645-ecd5-4ca5-a8b2-569179b19b6a" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:12:18.724100 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.724063 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1d053645-ecd5-4ca5-a8b2-569179b19b6a" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:12:18.724908 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.724884 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-kube-api-access-rdttm" (OuterVolumeSpecName: "kube-api-access-rdttm") pod "1d053645-ecd5-4ca5-a8b2-569179b19b6a" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a"). InnerVolumeSpecName "kube-api-access-rdttm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:12:18.725253 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.725222 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1d053645-ecd5-4ca5-a8b2-569179b19b6a" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:12:18.725726 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.725679 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1d053645-ecd5-4ca5-a8b2-569179b19b6a" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:12:18.725867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.725806 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1d053645-ecd5-4ca5-a8b2-569179b19b6a" (UID: "1d053645-ecd5-4ca5-a8b2-569179b19b6a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:12:18.739196 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.739159 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:18.793743 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.793702 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:18.824053 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.824019 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-bound-sa-token\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:18.824053 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.824045 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-image-registry-private-configuration\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:18.824053 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.824054 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-trusted-ca\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:18.824335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.824069 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-certificates\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:18.824335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.824086 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d053645-ecd5-4ca5-a8b2-569179b19b6a-installation-pull-secrets\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:18.824335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.824101 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdttm\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-kube-api-access-rdttm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:18.849249 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.848995 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:18.973248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:18.973219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:19.025561 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.025524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:12:19.029092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.029023 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 12:12:19.038893 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.038867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f44d8c-0fac-49cf-8e58-e8e33a638319-metrics-certs\") pod \"network-metrics-daemon-jqth5\" (UID: \"c5f44d8c-0fac-49cf-8e58-e8e33a638319\") " pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:12:19.126603 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.126559 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:12:19.126803 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.126644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:12:19.129055 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.129029 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 12:12:19.129187 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.129060 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 12:12:19.140290 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.140261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/426a4044-78b3-421b-abd3-85f56961434a-original-pull-secret\") pod \"global-pull-secret-syncer-xmpjs\" (UID: \"426a4044-78b3-421b-abd3-85f56961434a\") " pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:12:19.140457 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.140265 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 12:12:19.149552 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.149516 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:19.151198 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.151169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzhr\" (UniqueName: \"kubernetes.io/projected/9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d-kube-api-access-fhzhr\") pod \"network-check-target-nf86x\" (UID: \"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d\") " pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:12:19.171749 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.171705 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2w9zn\"" Jul 21 12:12:19.176517 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.176492 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8v447\"" Jul 21 12:12:19.180318 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.180288 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jqth5" Jul 21 12:12:19.185582 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.185554 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:12:19.347967 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.347857 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:19.360888 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.360852 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xmpjs" Jul 21 12:12:19.448276 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.448238 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:19.556858 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.556827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:19.613389 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.613299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:19.684646 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.684620 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f95845fff-stkqs" Jul 21 12:12:19.716610 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.716581 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f95845fff-stkqs"] Jul 21 12:12:19.720653 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.720626 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f95845fff-stkqs"] Jul 21 12:12:19.832605 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:19.832550 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d053645-ecd5-4ca5-a8b2-569179b19b6a-registry-tls\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:12:20.178086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:20.177998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:20.358775 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:20.358525 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d053645-ecd5-4ca5-a8b2-569179b19b6a" path="/var/lib/kubelet/pods/1d053645-ecd5-4ca5-a8b2-569179b19b6a/volumes" Jul 21 12:12:20.520924 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:20.520890 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:20.613462 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:20.613285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:20.767018 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:20.766987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:21.044122 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.043629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:12:21.046489 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.046458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a413545d-fdc9-45c1-b1b9-79cb9965cfa7-metrics-tls\") pod \"dns-default-nbhbz\" (UID: \"a413545d-fdc9-45c1-b1b9-79cb9965cfa7\") " pod="openshift-dns/dns-default-nbhbz" Jul 21 12:12:21.081494 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.081460 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nf86x"] Jul 21 12:12:21.302172 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.302134 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fnnf7\"" Jul 21 12:12:21.310928 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.310894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nbhbz" Jul 21 12:12:21.477760 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.477723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:21.569706 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.569633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:21.690758 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.689762 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv"] Jul 21 12:12:21.691337 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.691304 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:21.695365 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.695346 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" Jul 21 12:12:21.698347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.698300 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 12:12:21.698466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.698424 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wqd6p\"" Jul 21 12:12:21.710582 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.710525 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv"] Jul 21 12:12:21.749243 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.749213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/564e8037-d7c8-4a4d-af58-a02ba80792be-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-658pv\" (UID: \"564e8037-d7c8-4a4d-af58-a02ba80792be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" Jul 21 12:12:21.760202 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:21.760167 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6b03cf_32dc_4f8a_b5ed_df8f1cec6c7d.slice/crio-4359c549353ffe71cc8113395488f4b228dc1887e7d60e925071b684630be966 WatchSource:0}: Error finding container 4359c549353ffe71cc8113395488f4b228dc1887e7d60e925071b684630be966: Status 404 returned error can't find the container with id 4359c549353ffe71cc8113395488f4b228dc1887e7d60e925071b684630be966 Jul 21 12:12:21.788862 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.788769 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:21.850020 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.849722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/564e8037-d7c8-4a4d-af58-a02ba80792be-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-658pv\" (UID: \"564e8037-d7c8-4a4d-af58-a02ba80792be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" Jul 21 12:12:21.853249 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.853062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/564e8037-d7c8-4a4d-af58-a02ba80792be-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-658pv\" (UID: \"564e8037-d7c8-4a4d-af58-a02ba80792be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" Jul 21 12:12:21.888480 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.888390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:21.912458 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.912412 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jqth5"] Jul 21 12:12:21.933533 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:21.933506 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xmpjs"] Jul 21 12:12:22.007257 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:22.007131 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" Jul 21 12:12:22.170337 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:22.170253 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426a4044_78b3_421b_abd3_85f56961434a.slice/crio-f1f71e4b7e82fa9866bd47da4c958178daa39264e024a6b81269416530ed1ae7 WatchSource:0}: Error finding container f1f71e4b7e82fa9866bd47da4c958178daa39264e024a6b81269416530ed1ae7: Status 404 returned error can't find the container with id f1f71e4b7e82fa9866bd47da4c958178daa39264e024a6b81269416530ed1ae7 Jul 21 12:12:22.170903 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:22.170880 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f44d8c_0fac_49cf_8e58_e8e33a638319.slice/crio-58ad9ec78e6fcce4b55a863532ba3cc60121f41adbf5e0862cfa3f53832f9cb6 WatchSource:0}: Error finding container 58ad9ec78e6fcce4b55a863532ba3cc60121f41adbf5e0862cfa3f53832f9cb6: Status 404 returned error can't find the container with id 58ad9ec78e6fcce4b55a863532ba3cc60121f41adbf5e0862cfa3f53832f9cb6 Jul 21 12:12:22.696381 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:22.696301 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqth5" event={"ID":"c5f44d8c-0fac-49cf-8e58-e8e33a638319","Type":"ContainerStarted","Data":"58ad9ec78e6fcce4b55a863532ba3cc60121f41adbf5e0862cfa3f53832f9cb6"} Jul 21 12:12:22.697563 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:22.697520 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xmpjs" event={"ID":"426a4044-78b3-421b-abd3-85f56961434a","Type":"ContainerStarted","Data":"f1f71e4b7e82fa9866bd47da4c958178daa39264e024a6b81269416530ed1ae7"} Jul 21 12:12:22.698901 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:22.698871 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nf86x" event={"ID":"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d","Type":"ContainerStarted","Data":"4359c549353ffe71cc8113395488f4b228dc1887e7d60e925071b684630be966"} Jul 21 12:12:22.866075 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:22.866044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:22.970980 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:22.970890 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:24.005684 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.005583 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nbhbz"] Jul 21 12:12:24.010454 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:24.010422 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda413545d_fdc9_45c1_b1b9_79cb9965cfa7.slice/crio-fe230354956911ec39b50db58b30e02b256f278993926aa4c86533edfa1f8f31 WatchSource:0}: Error finding container fe230354956911ec39b50db58b30e02b256f278993926aa4c86533edfa1f8f31: Status 404 returned error can't find the container with id fe230354956911ec39b50db58b30e02b256f278993926aa4c86533edfa1f8f31 Jul 21 12:12:24.027821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.027712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv"] Jul 21 12:12:24.033231 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:24.033193 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564e8037_d7c8_4a4d_af58_a02ba80792be.slice/crio-7bbfda44d850e790821db6650180e2ecf64ee999f3e987fc0409728e6974de4c WatchSource:0}: Error finding container 7bbfda44d850e790821db6650180e2ecf64ee999f3e987fc0409728e6974de4c: Status 404 returned error can't find the container with id 7bbfda44d850e790821db6650180e2ecf64ee999f3e987fc0409728e6974de4c Jul 21 12:12:24.708127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.708085 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxw7m" event={"ID":"e9cdf497-3071-44fc-8a5e-9dbf895ba5db","Type":"ContainerStarted","Data":"c2371a322b43a93cc7d39a83e65fbfbf05b583695fef3f62a21fb73354864e00"} Jul 21 12:12:24.710141 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.710081 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" event={"ID":"2900ce51-ea8b-45fd-8f32-5f9ad5d563ef","Type":"ContainerStarted","Data":"81f2ac37e2cd0b245e2eece265ad7c1af364b56ee0006bab87a865f6fdf5b02a"} Jul 21 12:12:24.710524 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.710411 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:24.712710 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.712588 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" event={"ID":"564e8037-d7c8-4a4d-af58-a02ba80792be","Type":"ContainerStarted","Data":"7bbfda44d850e790821db6650180e2ecf64ee999f3e987fc0409728e6974de4c"} Jul 21 12:12:24.712710 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.712664 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" Jul 21 12:12:24.714481 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.714457 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" event={"ID":"e936553e-e7fe-415f-ae4a-d7a21b5bff69","Type":"ContainerStarted","Data":"00d3348a8e0c43b043b0cc1f85786c70be7ba8092b4ed734aec11d621ed38924"} Jul 21 12:12:24.715761 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.715727 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbhbz" event={"ID":"a413545d-fdc9-45c1-b1b9-79cb9965cfa7","Type":"ContainerStarted","Data":"fe230354956911ec39b50db58b30e02b256f278993926aa4c86533edfa1f8f31"} Jul 21 12:12:24.727592 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.727517 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dd99bffc-k24jv" podStartSLOduration=2.170146908 podStartE2EDuration="7.727500654s" totalImagesPullingTime="5.557353746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:24.726372368 +0000 UTC m=+70.895723645" watchObservedRunningTime="2026-07-21 12:12:24.727500654 +0000 UTC m=+70.896851929" Jul 21 12:12:24.759351 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:24.759293 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9dffdfb-z8862" podStartSLOduration=2.247949958 podStartE2EDuration="7.759275705s" totalImagesPullingTime="5.511325747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:24.758385485 +0000 UTC m=+70.927736762" watchObservedRunningTime="2026-07-21 12:12:24.759275705 +0000 UTC m=+70.928626975" Jul 21 12:12:25.725001 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:25.724489 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqth5" event={"ID":"c5f44d8c-0fac-49cf-8e58-e8e33a638319","Type":"ContainerStarted","Data":"a78e495b206954978e5be7ce1190a58ed29a61cdbc03f35df523d0e72e1f07da"} Jul 21 12:12:25.725001 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:25.724551 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jqth5" event={"ID":"c5f44d8c-0fac-49cf-8e58-e8e33a638319","Type":"ContainerStarted","Data":"590aa64d5bf6f749da58072537cdf3ee72cb190c08348d53d6711a1cfb0bebbe"} Jul 21 12:12:25.741857 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:25.741151 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jqth5" podStartSLOduration=68.089588703 podStartE2EDuration="1m10.741127899s" totalImagesPullingTime="2.651539196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:25.740642896 +0000 UTC m=+71.909994175" watchObservedRunningTime="2026-07-21 12:12:25.741127899 +0000 UTC m=+71.910479179" Jul 21 12:12:25.819408 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:25.819310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:25.923015 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:25.922910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:25.979056 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:25.978917 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:26.875049 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:26.875014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:26.979179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:26.979149 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:28.793471 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.793439 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b65b5d99f-zmqwr"] Jul 21 12:12:28.797618 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.797596 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:28.800606 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.800585 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 12:12:28.800606 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.800594 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 12:12:28.801323 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.801306 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 12:12:28.801439 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.801311 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 12:12:28.801565 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.801548 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 12:12:28.801628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.801603 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kkd5b\"" Jul 21 12:12:28.801687 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.801624 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 12:12:28.801744 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.801548 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 12:12:28.806864 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.806844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b65b5d99f-zmqwr"] Jul 21 12:12:28.912942 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.912906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzfcs\" (UniqueName: \"kubernetes.io/projected/7af9e904-d675-4434-a80d-f7368535f3ad-kube-api-access-kzfcs\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:28.913151 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.912991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-console-config\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:28.913151 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.913052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-service-ca\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:28.913151 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.913101 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-serving-cert\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:28.913151 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.913135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-oauth-serving-cert\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:28.913356 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.913164 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-oauth-config\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:28.989615 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:28.989577 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:29.013557 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.013518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-console-config\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.013720 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.013564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-service-ca\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.013720 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.013703 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-serving-cert\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.013843 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.013753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-oauth-serving-cert\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.013843 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.013784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-oauth-config\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.013973 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.013868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzfcs\" (UniqueName: \"kubernetes.io/projected/7af9e904-d675-4434-a80d-f7368535f3ad-kube-api-access-kzfcs\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.014495 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.014447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-service-ca\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.014593 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.014494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-console-config\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.014593 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.014495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-oauth-serving-cert\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.017097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.017075 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-oauth-config\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.017193 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.017154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-serving-cert\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.022628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.022606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzfcs\" (UniqueName: \"kubernetes.io/projected/7af9e904-d675-4434-a80d-f7368535f3ad-kube-api-access-kzfcs\") pod \"console-b65b5d99f-zmqwr\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:29.109063 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:29.108903 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:31.219898 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.219816 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b65b5d99f-zmqwr"] Jul 21 12:12:31.225478 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:31.225434 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af9e904_d675_4434_a80d_f7368535f3ad.slice/crio-041a6178480a51ad117a4ae8f1db3d37dda05e0e1fe12d7347f732b59d5ac184 WatchSource:0}: Error finding container 041a6178480a51ad117a4ae8f1db3d37dda05e0e1fe12d7347f732b59d5ac184: Status 404 returned error can't find the container with id 041a6178480a51ad117a4ae8f1db3d37dda05e0e1fe12d7347f732b59d5ac184 Jul 21 12:12:31.384334 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.384296 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:31.443760 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.443730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:31.683528 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.683441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:31.747720 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.747689 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbhbz" event={"ID":"a413545d-fdc9-45c1-b1b9-79cb9965cfa7","Type":"ContainerStarted","Data":"4ea33e06d54f22a86ea46593c357b84c031e6a8b57b4308491eaa845a5594571"} Jul 21 12:12:31.747922 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.747772 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbhbz" event={"ID":"a413545d-fdc9-45c1-b1b9-79cb9965cfa7","Type":"ContainerStarted","Data":"6c2942f80489a268119d68432b779a9e81dbce097928984093f648cd7fcebcc8"} Jul 21 12:12:31.747922 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.747800 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nbhbz" Jul 21 12:12:31.749127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.749096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b65b5d99f-zmqwr" event={"ID":"7af9e904-d675-4434-a80d-f7368535f3ad","Type":"ContainerStarted","Data":"041a6178480a51ad117a4ae8f1db3d37dda05e0e1fe12d7347f732b59d5ac184"} Jul 21 12:12:31.751068 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.751047 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kxw7m" event={"ID":"e9cdf497-3071-44fc-8a5e-9dbf895ba5db","Type":"ContainerStarted","Data":"a0ca97ced79cfb4db5618dbb744cab3cd5f2875e0e2cf677767a558f9cc2d969"} Jul 21 12:12:31.752623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.752597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xmpjs" event={"ID":"426a4044-78b3-421b-abd3-85f56961434a","Type":"ContainerStarted","Data":"66dac715591cb7eee071cabaeb1ccb2213eac256e5d6d9a49d5e2536fdf5c6d3"} Jul 21 12:12:31.753942 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.753922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" event={"ID":"564e8037-d7c8-4a4d-af58-a02ba80792be","Type":"ContainerStarted","Data":"b9e2ad6264e6291015fce5e2b114f46ad3e240f60b62cde5535755a171a8207f"} Jul 21 12:12:31.754082 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.754064 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" Jul 21 12:12:31.755448 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.755423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nf86x" event={"ID":"9d6b03cf-32dc-4f8a-b5ed-df8f1cec6c7d","Type":"ContainerStarted","Data":"a8a3a5385aabc22164ce8dd5ac5138ff286eb524af4516c11b70da13d6b53efc"} Jul 21 12:12:31.755551 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.755536 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:12:31.759066 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.759050 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" Jul 21 12:12:31.765305 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.765265 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nbhbz" podStartSLOduration=35.734585131 podStartE2EDuration="42.765253781s" totalImagesPullingTime="7.03066865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:31.764480106 +0000 UTC m=+77.933831383" watchObservedRunningTime="2026-07-21 12:12:31.765253781 +0000 UTC m=+77.934605034" Jul 21 12:12:31.780614 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.780559 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nf86x" podStartSLOduration=67.489238369 podStartE2EDuration="1m16.780539133s" totalImagesPullingTime="9.291300764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:31.779524766 +0000 UTC m=+77.948876065" watchObservedRunningTime="2026-07-21 12:12:31.780539133 +0000 UTC m=+77.949890432" Jul 21 12:12:31.794504 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.794439 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xmpjs" podStartSLOduration=67.916619462 podStartE2EDuration="1m16.794420694s" totalImagesPullingTime="8.877801232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:31.793452762 +0000 UTC m=+77.962804030" watchObservedRunningTime="2026-07-21 12:12:31.794420694 +0000 UTC m=+77.963771971" Jul 21 12:12:31.808327 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.808282 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-658pv" podStartSLOduration=3.799099444 podStartE2EDuration="10.80826581s" totalImagesPullingTime="7.009166366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:31.807784525 +0000 UTC m=+77.977135798" watchObservedRunningTime="2026-07-21 12:12:31.80826581 +0000 UTC m=+77.977617085" Jul 21 12:12:31.828159 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:31.828099 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kxw7m" podStartSLOduration=17.938588775 podStartE2EDuration="37.828075591s" totalImagesPullingTime="19.889486816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 12:11:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:31.82719312 +0000 UTC m=+77.996544401" watchObservedRunningTime="2026-07-21 12:12:31.828075591 +0000 UTC m=+77.997426867" Jul 21 12:12:32.705986 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.705936 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:32.750392 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.750360 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj"] Jul 21 12:12:32.753602 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.753578 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.756936 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.756910 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 12:12:32.756936 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.756924 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 12:12:32.757152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.756979 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7rr64\"" Jul 21 12:12:32.757152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.757019 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 12:12:32.757152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.757108 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 12:12:32.757301 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.757293 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 12:12:32.761711 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.761690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj"] Jul 21 12:12:32.781968 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.781918 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:32.845681 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.845344 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d72b58-3205-47f5-87d2-96427d9c667f-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.845681 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.845658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d72b58-3205-47f5-87d2-96427d9c667f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.845937 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.845805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d72b58-3205-47f5-87d2-96427d9c667f-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.845937 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.845834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ptz\" (UniqueName: \"kubernetes.io/projected/b3d72b58-3205-47f5-87d2-96427d9c667f-kube-api-access-h2ptz\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.885218 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.885169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:32.946929 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.946898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d72b58-3205-47f5-87d2-96427d9c667f-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.947124 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.946963 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d72b58-3205-47f5-87d2-96427d9c667f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.947124 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.946998 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d72b58-3205-47f5-87d2-96427d9c667f-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.947124 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.947014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ptz\" (UniqueName: \"kubernetes.io/projected/b3d72b58-3205-47f5-87d2-96427d9c667f-kube-api-access-h2ptz\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.947766 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.947740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d72b58-3205-47f5-87d2-96427d9c667f-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.949755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.949732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d72b58-3205-47f5-87d2-96427d9c667f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.949755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.949751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d72b58-3205-47f5-87d2-96427d9c667f-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:32.956150 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:32.956078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ptz\" (UniqueName: \"kubernetes.io/projected/b3d72b58-3205-47f5-87d2-96427d9c667f-kube-api-access-h2ptz\") pod \"prometheus-operator-5945b5cfd-8gqtj\" (UID: \"b3d72b58-3205-47f5-87d2-96427d9c667f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:33.066031 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:33.065999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" Jul 21 12:12:33.208082 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:33.208004 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj"] Jul 21 12:12:33.211865 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:33.211834 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d72b58_3205_47f5_87d2_96427d9c667f.slice/crio-b8a66fb5db8f5e08b4c3895a6b7a58f8ca341f1c680054ec0bf1210a90a093aa WatchSource:0}: Error finding container b8a66fb5db8f5e08b4c3895a6b7a58f8ca341f1c680054ec0bf1210a90a093aa: Status 404 returned error can't find the container with id b8a66fb5db8f5e08b4c3895a6b7a58f8ca341f1c680054ec0bf1210a90a093aa Jul 21 12:12:33.648616 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:33.648573 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:33.748497 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:33.748460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:33.764568 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:33.764531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" event={"ID":"b3d72b58-3205-47f5-87d2-96427d9c667f","Type":"ContainerStarted","Data":"b8a66fb5db8f5e08b4c3895a6b7a58f8ca341f1c680054ec0bf1210a90a093aa"} Jul 21 12:12:34.385628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:34.385593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:34.487228 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:34.487197 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:34.770337 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:34.770294 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b65b5d99f-zmqwr" event={"ID":"7af9e904-d675-4434-a80d-f7368535f3ad","Type":"ContainerStarted","Data":"1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b"} Jul 21 12:12:34.787801 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:34.787748 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b65b5d99f-zmqwr" podStartSLOduration=3.625783729 podStartE2EDuration="6.787733207s" totalImagesPullingTime="3.161949478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:34.786669895 +0000 UTC m=+80.956021185" watchObservedRunningTime="2026-07-21 12:12:34.787733207 +0000 UTC m=+80.957084482" Jul 21 12:12:35.698204 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:35.698056 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:35.775719 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:35.775685 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" event={"ID":"b3d72b58-3205-47f5-87d2-96427d9c667f","Type":"ContainerStarted","Data":"c1f01f360595e5c6675aabac14227770fbcd5b10e617045517b73e2a19611584"} Jul 21 12:12:35.776167 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:35.775745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" event={"ID":"b3d72b58-3205-47f5-87d2-96427d9c667f","Type":"ContainerStarted","Data":"b6dd8ea92398d6b534198089328de39b4496c11a6ec7c13e21da68c1dca4e309"} Jul 21 12:12:35.792444 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:35.792390 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8gqtj" podStartSLOduration=1.5846412939999999 podStartE2EDuration="3.792373074s" totalImagesPullingTime="2.20773178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:35.791844818 +0000 UTC m=+81.961196107" watchObservedRunningTime="2026-07-21 12:12:35.792373074 +0000 UTC m=+81.961724351" Jul 21 12:12:37.889168 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:37.889129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.030476 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.030441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.110690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.110646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.131239 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.131201 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lnx9q"] Jul 21 12:12:38.145684 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.145603 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t2w2d"] Jul 21 12:12:38.145981 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.145941 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.148559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.148533 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 12:12:38.149009 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.148785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 12:12:38.149119 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.148859 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fkpgz\"" Jul 21 12:12:38.149607 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.149588 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 12:12:38.150089 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.150046 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.153006 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.152981 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 12:12:38.153166 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.153126 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lnx9q"] Jul 21 12:12:38.153328 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.153312 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 12:12:38.153441 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.153430 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cj58p\"" Jul 21 12:12:38.153610 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.153591 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 12:12:38.235292 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.235262 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.287771 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.287741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblsm\" (UniqueName: \"kubernetes.io/projected/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-api-access-rblsm\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.287988 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.287783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.287988 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.287816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-textfile\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.287988 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.287889 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-wtmp\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.287988 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.287921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.287988 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.287973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-sys\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288000 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcv6\" (UniqueName: \"kubernetes.io/projected/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-kube-api-access-bjcv6\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288068 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288128 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-tls\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288179 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-root\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.288255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-metrics-client-ca\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.288529 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.288282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.389252 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.389252 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-root\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-metrics-client-ca\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389303 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.389466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblsm\" (UniqueName: \"kubernetes.io/projected/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-api-access-rblsm\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.389466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-textfile\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389336 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-root\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389740 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:38.389474 2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 21 12:12:38.389740 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:38.389547 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-tls podName:b14c2e65-60cb-4354-9cb9-ff7c88a96db3 nodeName:}" failed. No retries permitted until 2026-07-21 12:12:38.889525357 +0000 UTC m=+85.058876633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-lnx9q" (UID: "b14c2e65-60cb-4354-9cb9-ff7c88a96db3") : secret "kube-state-metrics-tls" not found Jul 21 12:12:38.389740 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389595 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-wtmp\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389740 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-wtmp\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389740 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.389740 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-sys\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.389740 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcv6\" (UniqueName: \"kubernetes.io/projected/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-kube-api-access-bjcv6\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-textfile\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-sys\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.389925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-tls\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.390034 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.390055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-metrics-client-ca\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:38.390067 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 12:12:38.390111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.390085 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.390585 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:12:38.390122 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-tls podName:677b3c04-ae20-4b8b-9a30-1d06ce7e99cd nodeName:}" failed. No retries permitted until 2026-07-21 12:12:38.890104177 +0000 UTC m=+85.059455444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-tls") pod "node-exporter-t2w2d" (UID: "677b3c04-ae20-4b8b-9a30-1d06ce7e99cd") : secret "node-exporter-tls" not found Jul 21 12:12:38.390657 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.390581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.390801 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.390780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.391796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.391778 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.391940 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.391925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.397690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.397640 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcv6\" (UniqueName: \"kubernetes.io/projected/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-kube-api-access-bjcv6\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.397690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.397663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblsm\" (UniqueName: \"kubernetes.io/projected/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-api-access-rblsm\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.460162 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.460130 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.590145 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.590100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.690555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.690476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.788623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.788594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:38.892211 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.892182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.892577 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.892273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-tls\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.894531 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.894508 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677b3c04-ae20-4b8b-9a30-1d06ce7e99cd-node-exporter-tls\") pod \"node-exporter-t2w2d\" (UID: \"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd\") " pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:38.894618 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.894601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b14c2e65-60cb-4354-9cb9-ff7c88a96db3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lnx9q\" (UID: \"b14c2e65-60cb-4354-9cb9-ff7c88a96db3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:38.949401 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:38.949305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:39.060603 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.060569 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" Jul 21 12:12:39.065725 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.065699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2w2d" Jul 21 12:12:39.073614 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.073580 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:39.077625 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:39.077597 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677b3c04_ae20_4b8b_9a30_1d06ce7e99cd.slice/crio-9f3f6095bf6ecf50eeb916f27698bfbbdf4cf251676275090911dea46c1bb6f0 WatchSource:0}: Error finding container 9f3f6095bf6ecf50eeb916f27698bfbbdf4cf251676275090911dea46c1bb6f0: Status 404 returned error can't find the container with id 9f3f6095bf6ecf50eeb916f27698bfbbdf4cf251676275090911dea46c1bb6f0 Jul 21 12:12:39.109311 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.109266 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:39.109492 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.109326 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:39.115358 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.115335 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:39.164409 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.164382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:39.211115 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.211033 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lnx9q"] Jul 21 12:12:39.215240 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:39.215212 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14c2e65_60cb_4354_9cb9_ff7c88a96db3.slice/crio-168c83f3313d207dc58374c6f8d8eca69dbd1811f4d46fb12adcf3db901a5fa2 WatchSource:0}: Error finding container 168c83f3313d207dc58374c6f8d8eca69dbd1811f4d46fb12adcf3db901a5fa2: Status 404 returned error can't find the container with id 168c83f3313d207dc58374c6f8d8eca69dbd1811f4d46fb12adcf3db901a5fa2 Jul 21 12:12:39.689893 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.689868 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-644f4d8c7f-t5c6h" Jul 21 12:12:39.791585 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.791546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" event={"ID":"b14c2e65-60cb-4354-9cb9-ff7c88a96db3","Type":"ContainerStarted","Data":"168c83f3313d207dc58374c6f8d8eca69dbd1811f4d46fb12adcf3db901a5fa2"} Jul 21 12:12:39.792992 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.792961 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2w2d" event={"ID":"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd","Type":"ContainerStarted","Data":"9f3f6095bf6ecf50eeb916f27698bfbbdf4cf251676275090911dea46c1bb6f0"} Jul 21 12:12:39.798604 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.798581 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:12:39.886576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.886532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:39.987061 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:39.987027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:40.088064 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:40.088025 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:40.181795 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:40.181755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:40.799580 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:40.799513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" event={"ID":"b14c2e65-60cb-4354-9cb9-ff7c88a96db3","Type":"ContainerStarted","Data":"4a7a4cd2d463c2f6d074e0557ca4f63b78a92641f6550cd10aaa6420216e9920"} Jul 21 12:12:40.799580 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:40.799582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" event={"ID":"b14c2e65-60cb-4354-9cb9-ff7c88a96db3","Type":"ContainerStarted","Data":"c8c9a7809b8cc20a424148ea7e68e7273c3135796d7e3d805125da16a81b328d"} Jul 21 12:12:40.801672 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:40.801626 2559 generic.go:356] "Generic (PLEG): container finished" podID="677b3c04-ae20-4b8b-9a30-1d06ce7e99cd" containerID="55881f3b351ca096b61c7924426843c810536bc5f4a0465348d87ed93827522d" exitCode=0 Jul 21 12:12:40.801768 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:40.801701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2w2d" event={"ID":"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd","Type":"ContainerDied","Data":"55881f3b351ca096b61c7924426843c810536bc5f4a0465348d87ed93827522d"} Jul 21 12:12:40.909629 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:40.909566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:41.008411 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.008380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:41.108370 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.108335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:41.163929 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.163790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:41.222190 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.222160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:41.619000 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.618968 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:41.761448 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.761419 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nbhbz" Jul 21 12:12:41.807558 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.807523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" event={"ID":"b14c2e65-60cb-4354-9cb9-ff7c88a96db3","Type":"ContainerStarted","Data":"97e71ad5a07e9041d57b1ad335e7913740355d350ad6e57f8227f4c3ac6d7f73"} Jul 21 12:12:41.809802 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.809774 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2w2d" event={"ID":"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd","Type":"ContainerStarted","Data":"6624030f151cf394eaa70eedff757f29a5d5c113e1dba597bd740dee993771ab"} Jul 21 12:12:41.809982 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.809825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2w2d" event={"ID":"677b3c04-ae20-4b8b-9a30-1d06ce7e99cd","Type":"ContainerStarted","Data":"9272969e5ef00390ea7a9891cedc3542357f6cf4dc2b0fafb08fd12c89df085c"} Jul 21 12:12:41.825066 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.824977 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-lnx9q" podStartSLOduration=2.388638497 podStartE2EDuration="3.824942849s" totalImagesPullingTime="1.436304352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:41.824107465 +0000 UTC m=+87.993458766" watchObservedRunningTime="2026-07-21 12:12:41.824942849 +0000 UTC m=+87.994294127" Jul 21 12:12:41.842462 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:41.842412 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t2w2d" podStartSLOduration=2.851192709 podStartE2EDuration="3.842397376s" totalImagesPullingTime="910.864045ms" totalInitContainerRuntime="80.340622ms" isStatefulPod=true podCreationTimestamp="2026-07-21 12:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:41.84122798 +0000 UTC m=+88.010579255" watchObservedRunningTime="2026-07-21 12:12:41.842397376 +0000 UTC m=+88.011748652" Jul 21 12:12:42.884690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:42.884650 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9"] Jul 21 12:12:42.888134 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:42.888108 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:42.890409 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:42.890370 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 12:12:42.890536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:42.890457 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lfw8g\"" Jul 21 12:12:42.898642 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:42.898612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:42.898834 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:42.898816 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9"] Jul 21 12:12:42.952493 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:42.952458 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:43.021069 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.021027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:43.029255 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.029231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxm2s\" (UniqueName: \"kubernetes.io/projected/317fced9-b067-4e34-b9ba-02b6fb876f09-kube-api-access-wxm2s\") pod \"monitoring-plugin-7b7b7bc8bf-854t9\" (UID: \"317fced9-b067-4e34-b9ba-02b6fb876f09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:43.029358 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.029262 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/317fced9-b067-4e34-b9ba-02b6fb876f09-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-854t9\" (UID: \"317fced9-b067-4e34-b9ba-02b6fb876f09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:43.129782 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.129748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxm2s\" (UniqueName: \"kubernetes.io/projected/317fced9-b067-4e34-b9ba-02b6fb876f09-kube-api-access-wxm2s\") pod \"monitoring-plugin-7b7b7bc8bf-854t9\" (UID: \"317fced9-b067-4e34-b9ba-02b6fb876f09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:43.129782 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.129786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/317fced9-b067-4e34-b9ba-02b6fb876f09-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-854t9\" (UID: \"317fced9-b067-4e34-b9ba-02b6fb876f09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:43.132189 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.132168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/317fced9-b067-4e34-b9ba-02b6fb876f09-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-854t9\" (UID: \"317fced9-b067-4e34-b9ba-02b6fb876f09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:43.137490 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.137422 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxm2s\" (UniqueName: \"kubernetes.io/projected/317fced9-b067-4e34-b9ba-02b6fb876f09-kube-api-access-wxm2s\") pod \"monitoring-plugin-7b7b7bc8bf-854t9\" (UID: \"317fced9-b067-4e34-b9ba-02b6fb876f09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:43.199244 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.199204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:43.334395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.334238 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9"] Jul 21 12:12:43.337130 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:43.337104 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317fced9_b067_4e34_b9ba_02b6fb876f09.slice/crio-9738a01955ff3cc8840d2311b944ac5310a774d06ac9499944ee23a786c631fc WatchSource:0}: Error finding container 9738a01955ff3cc8840d2311b944ac5310a774d06ac9499944ee23a786c631fc: Status 404 returned error can't find the container with id 9738a01955ff3cc8840d2311b944ac5310a774d06ac9499944ee23a786c631fc Jul 21 12:12:43.823714 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:43.823681 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" event={"ID":"317fced9-b067-4e34-b9ba-02b6fb876f09","Type":"ContainerStarted","Data":"9738a01955ff3cc8840d2311b944ac5310a774d06ac9499944ee23a786c631fc"} Jul 21 12:12:44.356915 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.356854 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:44.359324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.359213 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:12:44.363597 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.363571 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.366324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366014 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 12:12:44.366324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366022 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 12:12:44.366324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366068 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 12:12:44.366324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366017 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 12:12:44.366324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366160 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 12:12:44.366324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366213 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 12:12:44.366720 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366426 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lkzmk\"" Jul 21 12:12:44.367108 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.366939 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 12:12:44.372217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.367400 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 12:12:44.372217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.367485 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 12:12:44.372217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.367621 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 12:12:44.372217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.367767 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 12:12:44.372217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.371169 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9ca42ekp7mlc2\"" Jul 21 12:12:44.372217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.371403 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 12:12:44.372217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.371656 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 12:12:44.372744 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.372709 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 12:12:44.373673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.373040 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 12:12:44.374651 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.374633 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:12:44.440310 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.440275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.440588 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.440562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.440732 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.440715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.440852 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.440836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441003 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.440988 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441120 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441233 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441485 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441469 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441588 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441720 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441835 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441821 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqft\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-kube-api-access-jzqft\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.441990 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.441973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.442112 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.442099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.442232 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.442217 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.442362 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.442339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.442486 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.442471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.442613 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.442598 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.442744 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.442729 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.442853 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.442838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543721 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqft\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-kube-api-access-jzqft\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.543976 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.543999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.544555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.545089 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.545089 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.545089 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.545089 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.545089 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.544930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.547998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.545886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.547998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.546226 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.547998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.547167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.547998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.547659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.547998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.547755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.551061 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.551035 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.551358 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.551333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.551358 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.551350 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.551488 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.551371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.552025 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.551818 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.552025 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.551883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.552025 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.552005 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.552440 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.552404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.553007 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.552984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.553425 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.553400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.554233 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.554211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.554573 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.554554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.556047 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.556019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqft\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-kube-api-access-jzqft\") pod \"prometheus-k8s-0\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.618313 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.618235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:44.681391 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.681095 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:12:44.731700 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.731668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:44.972405 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:44.972369 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:12:44.975878 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:12:44.975850 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5e372e_42e4_4405_80e8_ec5fc80f38cb.slice/crio-3aa90a231f69cec911ec3443612b1fa4a93096b7635dc979da7ca25440cb8645 WatchSource:0}: Error finding container 3aa90a231f69cec911ec3443612b1fa4a93096b7635dc979da7ca25440cb8645: Status 404 returned error can't find the container with id 3aa90a231f69cec911ec3443612b1fa4a93096b7635dc979da7ca25440cb8645 Jul 21 12:12:45.834583 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:45.834548 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" event={"ID":"317fced9-b067-4e34-b9ba-02b6fb876f09","Type":"ContainerStarted","Data":"610239cfe4284fe89aba29f9952020a1590ea85d9b63815744ec194bab88edc8"} Jul 21 12:12:45.835072 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:45.834717 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:45.836074 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:45.836049 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerStarted","Data":"3aa90a231f69cec911ec3443612b1fa4a93096b7635dc979da7ca25440cb8645"} Jul 21 12:12:45.840570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:45.840546 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" Jul 21 12:12:45.851132 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:45.851075 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-854t9" podStartSLOduration=2.310784796 podStartE2EDuration="3.851057429s" totalImagesPullingTime="1.540272633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:45.850747248 +0000 UTC m=+92.020098528" watchObservedRunningTime="2026-07-21 12:12:45.851057429 +0000 UTC m=+92.020408706" Jul 21 12:12:46.840756 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:46.840726 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" exitCode=0 Jul 21 12:12:46.841165 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:46.840821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003"} Jul 21 12:12:48.642413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:48.642378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:48.751144 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:48.751102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:48.843617 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:48.843581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:49.138680 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:49.138647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:49.241396 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:49.241361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:49.334554 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:49.334517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:50.859938 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:50.859901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerStarted","Data":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} Jul 21 12:12:50.860357 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:50.860018 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerStarted","Data":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} Jul 21 12:12:50.910656 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:50.910619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:50.992355 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:50.992319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:52.871939 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:52.871905 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerStarted","Data":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} Jul 21 12:12:52.872440 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:52.872003 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerStarted","Data":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} Jul 21 12:12:52.872440 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:52.872044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerStarted","Data":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} Jul 21 12:12:52.872440 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:52.872090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerStarted","Data":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} Jul 21 12:12:52.901252 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:52.901161 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.498414464 podStartE2EDuration="8.901142529s" totalImagesPullingTime="6.382800375s" totalInitContainerRuntime="19.92769ms" isStatefulPod=false podCreationTimestamp="2026-07-21 12:12:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:12:52.899141862 +0000 UTC m=+99.068493137" watchObservedRunningTime="2026-07-21 12:12:52.901142529 +0000 UTC m=+99.070493805" Jul 21 12:12:53.641869 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:53.641829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:53.742663 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:53.742629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:12:54.681714 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:12:54.681678 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:13:01.967166 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:01.967132 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b65b5d99f-zmqwr"] Jul 21 12:13:02.083003 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:02.082941 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:02.187413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:02.187380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:02.393250 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:02.393219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:02.762268 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:02.762239 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nf86x" Jul 21 12:13:03.650095 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:03.650060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:03.751942 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:03.751905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:10.622779 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:10.622748 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4jt2_680cbf07-e123-4188-8052-8d691df7d859/serve-healthcheck-canary/0.log" Jul 21 12:13:18.654467 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:18.654408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:18.710131 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:18.710099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:25.603927 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:25.603884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:25.703152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:25.703119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:25.759635 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:25.759597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:26.987605 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:26.987558 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b65b5d99f-zmqwr" podUID="7af9e904-d675-4434-a80d-f7368535f3ad" containerName="console" containerID="cri-o://1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b" gracePeriod=15 Jul 21 12:13:27.228686 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.228660 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:13:27.327804 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.327711 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-oauth-config\") pod \"7af9e904-d675-4434-a80d-f7368535f3ad\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " Jul 21 12:13:27.327804 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.327751 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-oauth-serving-cert\") pod \"7af9e904-d675-4434-a80d-f7368535f3ad\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " Jul 21 12:13:27.327804 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.327773 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-serving-cert\") pod \"7af9e904-d675-4434-a80d-f7368535f3ad\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " Jul 21 12:13:27.328130 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.327828 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzfcs\" (UniqueName: \"kubernetes.io/projected/7af9e904-d675-4434-a80d-f7368535f3ad-kube-api-access-kzfcs\") pod \"7af9e904-d675-4434-a80d-f7368535f3ad\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " Jul 21 12:13:27.328130 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.327855 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-service-ca\") pod \"7af9e904-d675-4434-a80d-f7368535f3ad\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " Jul 21 12:13:27.328130 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.327902 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-console-config\") pod \"7af9e904-d675-4434-a80d-f7368535f3ad\" (UID: \"7af9e904-d675-4434-a80d-f7368535f3ad\") " Jul 21 12:13:27.328362 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.328334 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7af9e904-d675-4434-a80d-f7368535f3ad" (UID: "7af9e904-d675-4434-a80d-f7368535f3ad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:13:27.328489 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.328456 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-service-ca" (OuterVolumeSpecName: "service-ca") pod "7af9e904-d675-4434-a80d-f7368535f3ad" (UID: "7af9e904-d675-4434-a80d-f7368535f3ad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:13:27.328489 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.328473 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-console-config" (OuterVolumeSpecName: "console-config") pod "7af9e904-d675-4434-a80d-f7368535f3ad" (UID: "7af9e904-d675-4434-a80d-f7368535f3ad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:13:27.330336 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.330309 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7af9e904-d675-4434-a80d-f7368535f3ad" (UID: "7af9e904-d675-4434-a80d-f7368535f3ad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:13:27.330453 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.330357 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af9e904-d675-4434-a80d-f7368535f3ad-kube-api-access-kzfcs" (OuterVolumeSpecName: "kube-api-access-kzfcs") pod "7af9e904-d675-4434-a80d-f7368535f3ad" (UID: "7af9e904-d675-4434-a80d-f7368535f3ad"). InnerVolumeSpecName "kube-api-access-kzfcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:13:27.330453 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.330367 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7af9e904-d675-4434-a80d-f7368535f3ad" (UID: "7af9e904-d675-4434-a80d-f7368535f3ad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:13:27.428796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.428756 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-oauth-config\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:13:27.428796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.428789 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-oauth-serving-cert\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:13:27.428796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.428802 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af9e904-d675-4434-a80d-f7368535f3ad-console-serving-cert\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:13:27.429062 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.428814 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzfcs\" (UniqueName: \"kubernetes.io/projected/7af9e904-d675-4434-a80d-f7368535f3ad-kube-api-access-kzfcs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:13:27.429062 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.428826 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-service-ca\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:13:27.429062 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:27.428837 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af9e904-d675-4434-a80d-f7368535f3ad-console-config\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:13:28.008999 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.008935 2559 generic.go:356] "Generic (PLEG): container finished" podID="7af9e904-d675-4434-a80d-f7368535f3ad" containerID="1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b" exitCode=0 Jul 21 12:13:28.009411 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.009032 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b65b5d99f-zmqwr" Jul 21 12:13:28.009411 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.009027 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b65b5d99f-zmqwr" event={"ID":"7af9e904-d675-4434-a80d-f7368535f3ad","Type":"ContainerDied","Data":"1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b"} Jul 21 12:13:28.009411 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.009187 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b65b5d99f-zmqwr" event={"ID":"7af9e904-d675-4434-a80d-f7368535f3ad","Type":"ContainerDied","Data":"041a6178480a51ad117a4ae8f1db3d37dda05e0e1fe12d7347f732b59d5ac184"} Jul 21 12:13:28.009411 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.009209 2559 scope.go:117] "RemoveContainer" containerID="1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b" Jul 21 12:13:28.017674 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.017660 2559 scope.go:117] "RemoveContainer" containerID="1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b" Jul 21 12:13:28.017979 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:13:28.017931 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b\": container with ID starting with 1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b not found: ID does not exist" containerID="1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b" Jul 21 12:13:28.018073 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.018039 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b"} err="failed to get container status \"1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b\": rpc error: code = NotFound desc = could not find container \"1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b\": container with ID starting with 1dc04dd7105b9eee90810ebdbd32b6f9111fb5e8627d05d0c5046805a92dcc6b not found: ID does not exist" Jul 21 12:13:28.029030 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.029004 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b65b5d99f-zmqwr"] Jul 21 12:13:28.033187 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.033168 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b65b5d99f-zmqwr"] Jul 21 12:13:28.357143 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:28.357062 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af9e904-d675-4434-a80d-f7368535f3ad" path="/var/lib/kubelet/pods/7af9e904-d675-4434-a80d-f7368535f3ad/volumes" Jul 21 12:13:44.681921 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:44.681883 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:13:44.700630 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:44.700602 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:13:45.086052 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:45.086026 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:13:54.215266 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:54.215232 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:13:54.273131 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:13:54.273097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:02.726859 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.726822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:14:02.727344 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.727240 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="prometheus" containerID="cri-o://bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" gracePeriod=600 Jul 21 12:14:02.727344 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.727264 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="thanos-sidecar" containerID="cri-o://69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" gracePeriod=600 Jul 21 12:14:02.727344 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.727322 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy" containerID="cri-o://7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" gracePeriod=600 Jul 21 12:14:02.727501 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.727278 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" gracePeriod=600 Jul 21 12:14:02.727501 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.727325 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-web" containerID="cri-o://431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" gracePeriod=600 Jul 21 12:14:02.727501 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.727287 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="config-reloader" containerID="cri-o://eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" gracePeriod=600 Jul 21 12:14:02.981195 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:02.981120 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019163 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-tls\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019205 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019229 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-2\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019252 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019276 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-web-config\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019291 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-db\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019313 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-kubelet-serving-ca-bundle\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019334 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-serving-certs-ca-bundle\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019349 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-1\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019365 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-grpc-tls\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019384 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-trusted-ca-bundle\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019407 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-kube-rbac-proxy\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019430 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-tls-assets\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019447 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-0\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019468 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019495 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-metrics-client-ca\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.019570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019518 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-thanos-prometheus-http-client-file\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.020537 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019805 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:14:03.020537 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019866 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzqft\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-kube-api-access-jzqft\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.020537 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019901 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-metrics-client-certs\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.020537 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.019944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config-out\") pod \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\" (UID: \"6c5e372e-42e4-4405-80e8-ec5fc80f38cb\") " Jul 21 12:14:03.020537 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.020216 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.020789 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.020647 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:14:03.021847 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.021713 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:14:03.022841 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.022744 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:14:03.022841 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.022813 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:14:03.023247 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.023223 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:14:03.023335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.023286 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:14:03.023513 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.023496 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 12:14:03.023814 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.023774 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.024228 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.024189 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.024563 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.024529 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config-out" (OuterVolumeSpecName: "config-out") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:14:03.027998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.027833 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.028363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.028332 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:14:03.028456 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.028376 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config" (OuterVolumeSpecName: "config") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.028594 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.028554 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.029419 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.029386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.029515 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.029476 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.029876 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.029847 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-kube-api-access-jzqft" (OuterVolumeSpecName: "kube-api-access-jzqft") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "kube-api-access-jzqft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:14:03.030017 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.029941 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.040417 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.040385 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-web-config" (OuterVolumeSpecName: "web-config") pod "6c5e372e-42e4-4405-80e8-ec5fc80f38cb" (UID: "6c5e372e-42e4-4405-80e8-ec5fc80f38cb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:14:03.120837 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120798 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-metrics-client-ca\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.120837 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120830 2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.120837 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120841 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzqft\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-kube-api-access-jzqft\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120850 2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-metrics-client-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120861 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config-out\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120869 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120878 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120887 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120896 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120905 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-web-config\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120913 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-db\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120922 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120931 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120940 2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-grpc-tls\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120963 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120971 2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-secret-kube-rbac-proxy\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120979 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-tls-assets\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120991 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.121106 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.120999 2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c5e372e-42e4-4405-80e8-ec5fc80f38cb-config\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:14:03.143044 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143012 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" exitCode=0 Jul 21 12:14:03.143044 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143037 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" exitCode=0 Jul 21 12:14:03.143044 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143045 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" exitCode=0 Jul 21 12:14:03.143044 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143051 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" exitCode=0 Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143056 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" exitCode=0 Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143062 2559 generic.go:356] "Generic (PLEG): container finished" podID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" exitCode=0 Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143113 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143219 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143234 2559 scope.go:117] "RemoveContainer" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143276 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} Jul 21 12:14:03.143363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.143305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c5e372e-42e4-4405-80e8-ec5fc80f38cb","Type":"ContainerDied","Data":"3aa90a231f69cec911ec3443612b1fa4a93096b7635dc979da7ca25440cb8645"} Jul 21 12:14:03.151113 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.150996 2559 scope.go:117] "RemoveContainer" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.159750 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.159731 2559 scope.go:117] "RemoveContainer" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.166837 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.166818 2559 scope.go:117] "RemoveContainer" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.167529 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.167508 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:14:03.172453 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.172430 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:14:03.174622 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.174607 2559 scope.go:117] "RemoveContainer" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.181082 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.181051 2559 scope.go:117] "RemoveContainer" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.194102 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.194078 2559 scope.go:117] "RemoveContainer" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.198166 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198143 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:14:03.198453 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198438 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af9e904-d675-4434-a80d-f7368535f3ad" containerName="console" Jul 21 12:14:03.198527 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198457 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af9e904-d675-4434-a80d-f7368535f3ad" containerName="console" Jul 21 12:14:03.198527 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198470 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="prometheus" Jul 21 12:14:03.198527 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198479 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="prometheus" Jul 21 12:14:03.198527 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198493 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy" Jul 21 12:14:03.198527 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198501 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy" Jul 21 12:14:03.198527 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198525 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="thanos-sidecar" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198535 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="thanos-sidecar" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198545 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="init-config-reloader" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198553 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="init-config-reloader" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198561 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-thanos" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198570 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-thanos" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198582 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="config-reloader" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198591 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="config-reloader" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198601 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-web" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198609 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-web" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198670 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="prometheus" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198681 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="config-reloader" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198691 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-thanos" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198700 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="thanos-sidecar" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198708 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198716 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" containerName="kube-rbac-proxy-web" Jul 21 12:14:03.198812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.198724 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af9e904-d675-4434-a80d-f7368535f3ad" containerName="console" Jul 21 12:14:03.201874 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.201858 2559 scope.go:117] "RemoveContainer" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.202282 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:14:03.202262 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": container with ID starting with f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb not found: ID does not exist" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.202346 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.202291 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} err="failed to get container status \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": rpc error: code = NotFound desc = could not find container \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": container with ID starting with f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb not found: ID does not exist" Jul 21 12:14:03.202346 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.202310 2559 scope.go:117] "RemoveContainer" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.202562 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:14:03.202543 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": container with ID starting with 7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a not found: ID does not exist" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.202599 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.202568 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} err="failed to get container status \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": rpc error: code = NotFound desc = could not find container \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": container with ID starting with 7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a not found: ID does not exist" Jul 21 12:14:03.202599 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.202585 2559 scope.go:117] "RemoveContainer" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.202825 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:14:03.202798 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": container with ID starting with 431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893 not found: ID does not exist" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.202864 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.202830 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} err="failed to get container status \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": rpc error: code = NotFound desc = could not find container \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": container with ID starting with 431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893 not found: ID does not exist" Jul 21 12:14:03.202864 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.202844 2559 scope.go:117] "RemoveContainer" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.203097 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:14:03.203077 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": container with ID starting with 69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57 not found: ID does not exist" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.203172 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203105 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} err="failed to get container status \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": rpc error: code = NotFound desc = could not find container \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": container with ID starting with 69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57 not found: ID does not exist" Jul 21 12:14:03.203172 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203127 2559 scope.go:117] "RemoveContainer" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.203356 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:14:03.203340 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": container with ID starting with eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe not found: ID does not exist" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.203400 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203360 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} err="failed to get container status \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": rpc error: code = NotFound desc = could not find container \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": container with ID starting with eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe not found: ID does not exist" Jul 21 12:14:03.203400 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203374 2559 scope.go:117] "RemoveContainer" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.203564 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:14:03.203549 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": container with ID starting with bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad not found: ID does not exist" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.203605 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203567 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} err="failed to get container status \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": rpc error: code = NotFound desc = could not find container \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": container with ID starting with bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad not found: ID does not exist" Jul 21 12:14:03.203605 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203585 2559 scope.go:117] "RemoveContainer" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.203772 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:14:03.203756 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": container with ID starting with 88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003 not found: ID does not exist" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.203825 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203774 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003"} err="failed to get container status \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": rpc error: code = NotFound desc = could not find container \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": container with ID starting with 88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003 not found: ID does not exist" Jul 21 12:14:03.203825 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203786 2559 scope.go:117] "RemoveContainer" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.204018 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.203999 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} err="failed to get container status \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": rpc error: code = NotFound desc = could not find container \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": container with ID starting with f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb not found: ID does not exist" Jul 21 12:14:03.204066 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204020 2559 scope.go:117] "RemoveContainer" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.204240 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204224 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.204300 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204237 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} err="failed to get container status \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": rpc error: code = NotFound desc = could not find container \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": container with ID starting with 7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a not found: ID does not exist" Jul 21 12:14:03.204300 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204260 2559 scope.go:117] "RemoveContainer" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.204544 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204512 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} err="failed to get container status \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": rpc error: code = NotFound desc = could not find container \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": container with ID starting with 431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893 not found: ID does not exist" Jul 21 12:14:03.204544 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204543 2559 scope.go:117] "RemoveContainer" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.204838 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204818 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} err="failed to get container status \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": rpc error: code = NotFound desc = could not find container \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": container with ID starting with 69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57 not found: ID does not exist" Jul 21 12:14:03.204915 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.204858 2559 scope.go:117] "RemoveContainer" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.205136 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205114 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} err="failed to get container status \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": rpc error: code = NotFound desc = could not find container \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": container with ID starting with eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe not found: ID does not exist" Jul 21 12:14:03.205206 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205137 2559 scope.go:117] "RemoveContainer" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.205358 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205341 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} err="failed to get container status \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": rpc error: code = NotFound desc = could not find container \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": container with ID starting with bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad not found: ID does not exist" Jul 21 12:14:03.205406 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205359 2559 scope.go:117] "RemoveContainer" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.205643 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205622 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003"} err="failed to get container status \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": rpc error: code = NotFound desc = could not find container \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": container with ID starting with 88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003 not found: ID does not exist" Jul 21 12:14:03.205690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205646 2559 scope.go:117] "RemoveContainer" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.205868 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205852 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} err="failed to get container status \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": rpc error: code = NotFound desc = could not find container \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": container with ID starting with f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb not found: ID does not exist" Jul 21 12:14:03.205909 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.205871 2559 scope.go:117] "RemoveContainer" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.206116 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206095 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} err="failed to get container status \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": rpc error: code = NotFound desc = could not find container \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": container with ID starting with 7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a not found: ID does not exist" Jul 21 12:14:03.206166 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206119 2559 scope.go:117] "RemoveContainer" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.206448 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206432 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 12:14:03.206448 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206443 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 12:14:03.206591 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206432 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} err="failed to get container status \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": rpc error: code = NotFound desc = could not find container \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": container with ID starting with 431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893 not found: ID does not exist" Jul 21 12:14:03.206591 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206481 2559 scope.go:117] "RemoveContainer" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.206591 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206518 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 12:14:03.206591 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206588 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 12:14:03.206788 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206751 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} err="failed to get container status \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": rpc error: code = NotFound desc = could not find container \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": container with ID starting with 69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57 not found: ID does not exist" Jul 21 12:14:03.206788 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.206772 2559 scope.go:117] "RemoveContainer" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.207024 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207008 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 12:14:03.207105 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207089 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9ca42ekp7mlc2\"" Jul 21 12:14:03.207206 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207175 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 12:14:03.207206 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207182 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} err="failed to get container status \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": rpc error: code = NotFound desc = could not find container \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": container with ID starting with eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe not found: ID does not exist" Jul 21 12:14:03.207206 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207202 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 12:14:03.207413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207229 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lkzmk\"" Jul 21 12:14:03.207413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207204 2559 scope.go:117] "RemoveContainer" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.207413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207327 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 12:14:03.207413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207348 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 12:14:03.207413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207328 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 12:14:03.207623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207414 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 12:14:03.207623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207542 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} err="failed to get container status \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": rpc error: code = NotFound desc = could not find container \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": container with ID starting with bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad not found: ID does not exist" Jul 21 12:14:03.207623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207566 2559 scope.go:117] "RemoveContainer" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.207767 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207725 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 12:14:03.207873 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207854 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003"} err="failed to get container status \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": rpc error: code = NotFound desc = could not find container \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": container with ID starting with 88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003 not found: ID does not exist" Jul 21 12:14:03.207940 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207874 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 12:14:03.207940 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.207876 2559 scope.go:117] "RemoveContainer" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.208212 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.208190 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} err="failed to get container status \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": rpc error: code = NotFound desc = could not find container \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": container with ID starting with f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb not found: ID does not exist" Jul 21 12:14:03.208318 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.208213 2559 scope.go:117] "RemoveContainer" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.208607 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.208582 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} err="failed to get container status \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": rpc error: code = NotFound desc = could not find container \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": container with ID starting with 7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a not found: ID does not exist" Jul 21 12:14:03.208696 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.208609 2559 scope.go:117] "RemoveContainer" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.208867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.208846 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} err="failed to get container status \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": rpc error: code = NotFound desc = could not find container \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": container with ID starting with 431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893 not found: ID does not exist" Jul 21 12:14:03.208972 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.208868 2559 scope.go:117] "RemoveContainer" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.209235 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.209194 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} err="failed to get container status \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": rpc error: code = NotFound desc = could not find container \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": container with ID starting with 69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57 not found: ID does not exist" Jul 21 12:14:03.209235 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.209228 2559 scope.go:117] "RemoveContainer" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.209555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.209529 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} err="failed to get container status \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": rpc error: code = NotFound desc = could not find container \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": container with ID starting with eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe not found: ID does not exist" Jul 21 12:14:03.209555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.209548 2559 scope.go:117] "RemoveContainer" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.209877 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.209802 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} err="failed to get container status \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": rpc error: code = NotFound desc = could not find container \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": container with ID starting with bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad not found: ID does not exist" Jul 21 12:14:03.209877 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.209827 2559 scope.go:117] "RemoveContainer" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.210119 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210067 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003"} err="failed to get container status \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": rpc error: code = NotFound desc = could not find container \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": container with ID starting with 88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003 not found: ID does not exist" Jul 21 12:14:03.210119 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210091 2559 scope.go:117] "RemoveContainer" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.210368 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210327 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} err="failed to get container status \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": rpc error: code = NotFound desc = could not find container \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": container with ID starting with f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb not found: ID does not exist" Jul 21 12:14:03.210368 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210346 2559 scope.go:117] "RemoveContainer" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.210660 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210620 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} err="failed to get container status \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": rpc error: code = NotFound desc = could not find container \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": container with ID starting with 7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a not found: ID does not exist" Jul 21 12:14:03.210660 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210648 2559 scope.go:117] "RemoveContainer" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.210826 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210808 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 12:14:03.211020 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.210939 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} err="failed to get container status \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": rpc error: code = NotFound desc = could not find container \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": container with ID starting with 431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893 not found: ID does not exist" Jul 21 12:14:03.211020 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.211007 2559 scope.go:117] "RemoveContainer" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.211347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.211305 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} err="failed to get container status \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": rpc error: code = NotFound desc = could not find container \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": container with ID starting with 69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57 not found: ID does not exist" Jul 21 12:14:03.211347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.211330 2559 scope.go:117] "RemoveContainer" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.211741 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.211711 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} err="failed to get container status \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": rpc error: code = NotFound desc = could not find container \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": container with ID starting with eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe not found: ID does not exist" Jul 21 12:14:03.211848 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.211744 2559 scope.go:117] "RemoveContainer" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.212056 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.212031 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} err="failed to get container status \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": rpc error: code = NotFound desc = could not find container \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": container with ID starting with bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad not found: ID does not exist" Jul 21 12:14:03.212157 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.212057 2559 scope.go:117] "RemoveContainer" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.212543 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.212387 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003"} err="failed to get container status \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": rpc error: code = NotFound desc = could not find container \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": container with ID starting with 88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003 not found: ID does not exist" Jul 21 12:14:03.212543 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.212410 2559 scope.go:117] "RemoveContainer" containerID="f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb" Jul 21 12:14:03.212766 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.212717 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb"} err="failed to get container status \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": rpc error: code = NotFound desc = could not find container \"f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb\": container with ID starting with f71b2e6c3e06fcc45b8f694cea9484e2adcccce20a27c3ba1f8941fcb06d15eb not found: ID does not exist" Jul 21 12:14:03.212766 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.212744 2559 scope.go:117] "RemoveContainer" containerID="7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a" Jul 21 12:14:03.213081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213058 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a"} err="failed to get container status \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": rpc error: code = NotFound desc = could not find container \"7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a\": container with ID starting with 7d9979ab806fe83c95c0ce7dd0940f86f11045bd1cfdce79dc1b65ee7602638a not found: ID does not exist" Jul 21 12:14:03.213179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213084 2559 scope.go:117] "RemoveContainer" containerID="431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893" Jul 21 12:14:03.213320 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213300 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893"} err="failed to get container status \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": rpc error: code = NotFound desc = could not find container \"431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893\": container with ID starting with 431999c9a1ac04f4d318b0a414dd515c23c0b121ba34cafbb4e92ab29885c893 not found: ID does not exist" Jul 21 12:14:03.213387 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213320 2559 scope.go:117] "RemoveContainer" containerID="69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57" Jul 21 12:14:03.213555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213534 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 12:14:03.213555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213543 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57"} err="failed to get container status \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": rpc error: code = NotFound desc = could not find container \"69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57\": container with ID starting with 69d5f684141b755607ad1790941d399b1ba734fb0810d7d12f52a506aa1b9f57 not found: ID does not exist" Jul 21 12:14:03.213688 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213559 2559 scope.go:117] "RemoveContainer" containerID="eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe" Jul 21 12:14:03.213808 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213788 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe"} err="failed to get container status \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": rpc error: code = NotFound desc = could not find container \"eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe\": container with ID starting with eb749a121c38d1057640cfd121f24e42545f6bf2c14352aa343cd997a8fb5abe not found: ID does not exist" Jul 21 12:14:03.213856 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.213808 2559 scope.go:117] "RemoveContainer" containerID="bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad" Jul 21 12:14:03.214041 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.214025 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad"} err="failed to get container status \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": rpc error: code = NotFound desc = could not find container \"bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad\": container with ID starting with bc3a38ab56e9e0a64dd9a116f62612105cf48ebcfa9d977ff635cde300690aad not found: ID does not exist" Jul 21 12:14:03.214093 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.214042 2559 scope.go:117] "RemoveContainer" containerID="88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003" Jul 21 12:14:03.214249 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.214232 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003"} err="failed to get container status \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": rpc error: code = NotFound desc = could not find container \"88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003\": container with ID starting with 88f23514767cfde953e85a21524d357ecbff74bd833796e85ab75d2290482003 not found: ID does not exist" Jul 21 12:14:03.219683 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.219594 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:14:03.221622 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221719 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221719 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221646 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221719 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221719 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1cbddd0-55dd-489d-acae-fbc5148404a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221888 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221888 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221888 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221888 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.221888 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.221935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1cbddd0-55dd-489d-acae-fbc5148404a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2rr\" (UniqueName: \"kubernetes.io/projected/f1cbddd0-55dd-489d-acae-fbc5148404a3-kube-api-access-rs2rr\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222083 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222110 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-config\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222163 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222459 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.222459 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.222238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.322913 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.322824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.322913 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.322862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.322913 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.322891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.322913 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.322915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323207 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323207 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323207 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1cbddd0-55dd-489d-acae-fbc5148404a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323375 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323375 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323375 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323375 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323375 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1cbddd0-55dd-489d-acae-fbc5148404a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323503 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2rr\" (UniqueName: \"kubernetes.io/projected/f1cbddd0-55dd-489d-acae-fbc5148404a3-kube-api-access-rs2rr\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-config\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323564 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.323627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.324067 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.323665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.324227 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.324171 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.324302 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.324226 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.325045 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.325018 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.325925 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.325901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1cbddd0-55dd-489d-acae-fbc5148404a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.326380 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.326036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.326380 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.326115 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.326530 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.326491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.327179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.326807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.327179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.326927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.327179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.327120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.327179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.327134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.327650 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.327626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1cbddd0-55dd-489d-acae-fbc5148404a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.327907 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.327880 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1cbddd0-55dd-489d-acae-fbc5148404a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.328234 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.328213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-config\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.328551 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.328527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.328647 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.328628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.329075 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.329058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1cbddd0-55dd-489d-acae-fbc5148404a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.331850 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.331832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2rr\" (UniqueName: \"kubernetes.io/projected/f1cbddd0-55dd-489d-acae-fbc5148404a3-kube-api-access-rs2rr\") pod \"prometheus-k8s-0\" (UID: \"f1cbddd0-55dd-489d-acae-fbc5148404a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.516169 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.516110 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:03.657409 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:03.657374 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 12:14:03.660267 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:14:03.660244 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cbddd0_55dd_489d_acae_fbc5148404a3.slice/crio-858257a569d1ddecdea39e87a7db35380cdc07bb5706b2f95fb5c871da387ddb WatchSource:0}: Error finding container 858257a569d1ddecdea39e87a7db35380cdc07bb5706b2f95fb5c871da387ddb: Status 404 returned error can't find the container with id 858257a569d1ddecdea39e87a7db35380cdc07bb5706b2f95fb5c871da387ddb Jul 21 12:14:04.149424 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:04.149388 2559 generic.go:356] "Generic (PLEG): container finished" podID="f1cbddd0-55dd-489d-acae-fbc5148404a3" containerID="6af9256e8b0d85244d242b5780e151902b325eabbb04ddbf77f160d6d1ce16e7" exitCode=0 Jul 21 12:14:04.149905 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:04.149451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerDied","Data":"6af9256e8b0d85244d242b5780e151902b325eabbb04ddbf77f160d6d1ce16e7"} Jul 21 12:14:04.149905 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:04.149500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerStarted","Data":"858257a569d1ddecdea39e87a7db35380cdc07bb5706b2f95fb5c871da387ddb"} Jul 21 12:14:04.361347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:04.361308 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c5e372e-42e4-4405-80e8-ec5fc80f38cb" path="/var/lib/kubelet/pods/6c5e372e-42e4-4405-80e8-ec5fc80f38cb/volumes" Jul 21 12:14:05.157244 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:05.157206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerStarted","Data":"d2df04e9f05bb3c8c167d20b8fcf9d3121373739a6d1c2c347118e2dac1d8b46"} Jul 21 12:14:05.157735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:05.157282 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerStarted","Data":"878311bd724dfc5bf3e32c937c60b9636ea377ab08ec3c8d5a56fab1dc362b27"} Jul 21 12:14:05.157735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:05.157331 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerStarted","Data":"dad62c4e8aa8b3e9d6b7e3d4ee960fd28c82b93cc71e684702b8eb2634138816"} Jul 21 12:14:05.157735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:05.157372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerStarted","Data":"93f77bb1a6067192a71e1240e4d8482dbcc0fbc1fbac3c44fc7616c8176e7ea5"} Jul 21 12:14:05.157735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:05.157421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerStarted","Data":"3dfae586b438c529456fbd99abe24d1e55c368a06a8de16491ce63f7b9c790fa"} Jul 21 12:14:05.157735 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:05.157455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1cbddd0-55dd-489d-acae-fbc5148404a3","Type":"ContainerStarted","Data":"fb279c10a4d215a9b09b5cea9737ce9ce01e2b5e57d87aa8d72fca32137c9a99"} Jul 21 12:14:05.186416 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:05.186363 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.167327371 podStartE2EDuration="2.186343237s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.015866ms" isStatefulPod=false podCreationTimestamp="2026-07-21 12:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:14:05.183771741 +0000 UTC m=+171.353123017" watchObservedRunningTime="2026-07-21 12:14:05.186343237 +0000 UTC m=+171.355694512" Jul 21 12:14:06.294534 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:06.294434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:06.347600 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:06.347571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:08.517287 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:08.517246 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:14:18.659193 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:18.659157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:18.730119 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:18.730081 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:18.785672 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:18.785636 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:18.892043 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:18.892008 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:18.951731 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:18.951653 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:19.727075 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:19.727039 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:25.172741 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:25.172708 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:29.336619 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:29.336584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:29.436161 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:29.436129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:29.490119 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:29.490082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:40.724082 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:40.724049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:40.780022 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:40.779991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:14:56.802906 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:14:56.802875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:15:03.517347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:15:03.517305 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:15:03.532853 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:15:03.532825 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:15:04.387555 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:15:04.387531 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 12:15:33.468258 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:15:33.468226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:15:33.528732 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:15:33.528705 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:15:53.940620 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:15:53.940537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:16:14.215770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:16:14.215742 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:16:14.219043 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:16:14.219021 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 12:16:14.219588 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:16:14.219573 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:17:28.050941 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:17:28.050863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:17:30.175070 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:17:30.175031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:17:59.876812 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:17:59.876777 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:17:59.931821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:17:59.931790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:18:19.483927 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:18:19.483892 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:31.729035 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:31.728939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:31.797217 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:31.797184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:40.464868 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:40.464834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:53.470214 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:53.470175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:54.894318 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:54.894285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:57.090855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:57.090821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:57.190549 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:57.190517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:58.929536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:58.929495 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:20:59.080532 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:20:59.080494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:00.617177 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:00.617142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:01.340583 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:01.340556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:01.535927 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:01.535898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:04.032487 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:04.032455 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:08.323846 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:08.323812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:08.383527 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:08.383498 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:08.930051 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:08.930021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:12.806056 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:12.806021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:12.905982 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:12.905921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:13.641720 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:13.641683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:14.242448 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:14.242417 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:21:14.243578 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:14.243559 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:21:18.618124 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:18.618078 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:18.719868 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:18.719840 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:20.350516 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:20.350485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:32.420681 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:32.420619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:32.512087 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:32.512056 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.007370 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.007337 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.375585 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.375491 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.468539 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.468508 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.517320 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.517263 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn"] Jul 21 12:21:33.520810 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.520785 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.523480 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.523458 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 12:21:33.523676 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.523659 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 12:21:33.523904 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.523881 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b5mqv\"" Jul 21 12:21:33.524081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.523971 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 12:21:33.524806 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.524785 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 12:21:33.539092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.539065 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn"] Jul 21 12:21:33.613175 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.613139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cm9k\" (UniqueName: \"kubernetes.io/projected/4506e84a-8330-468d-a823-549b20f8cd0d-kube-api-access-2cm9k\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.613347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.613200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4506e84a-8330-468d-a823-549b20f8cd0d-tmp\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.613347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.613254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4506e84a-8330-468d-a823-549b20f8cd0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.613347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.613324 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4506e84a-8330-468d-a823-549b20f8cd0d-manifests\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.613456 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.613349 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4506e84a-8330-468d-a823-549b20f8cd0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.650962 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.650873 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.714420 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.714380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4506e84a-8330-468d-a823-549b20f8cd0d-manifests\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.714420 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.714423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4506e84a-8330-468d-a823-549b20f8cd0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.714693 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.714446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cm9k\" (UniqueName: \"kubernetes.io/projected/4506e84a-8330-468d-a823-549b20f8cd0d-kube-api-access-2cm9k\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.714693 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.714508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4506e84a-8330-468d-a823-549b20f8cd0d-tmp\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.714693 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.714527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4506e84a-8330-468d-a823-549b20f8cd0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.714921 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.714886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4506e84a-8330-468d-a823-549b20f8cd0d-manifests\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.716925 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.716895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4506e84a-8330-468d-a823-549b20f8cd0d-tmp\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.717064 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.716977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4506e84a-8330-468d-a823-549b20f8cd0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.717121 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.717087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4506e84a-8330-468d-a823-549b20f8cd0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.723893 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.723871 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cm9k\" (UniqueName: \"kubernetes.io/projected/4506e84a-8330-468d-a823-549b20f8cd0d-kube-api-access-2cm9k\") pod \"opendatahub-operator-controller-manager-58f85c6b8d-5qdxn\" (UID: \"4506e84a-8330-468d-a823-549b20f8cd0d\") " pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.762473 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.762438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.830804 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.830757 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:33.850716 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.850680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.955887 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.955852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:33.973283 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.973262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn"] Jul 21 12:21:33.976280 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:21:33.976254 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4506e84a_8330_468d_a823_549b20f8cd0d.slice/crio-b7550045c0aee1292b7b794570a1afa4771eff47dd39b43ec53a00c8b26dda95 WatchSource:0}: Error finding container b7550045c0aee1292b7b794570a1afa4771eff47dd39b43ec53a00c8b26dda95: Status 404 returned error can't find the container with id b7550045c0aee1292b7b794570a1afa4771eff47dd39b43ec53a00c8b26dda95 Jul 21 12:21:33.978069 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:33.978052 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:21:34.053226 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:34.053192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:34.328346 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:34.328313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:34.784002 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:34.783940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" event={"ID":"4506e84a-8330-468d-a823-549b20f8cd0d","Type":"ContainerStarted","Data":"b7550045c0aee1292b7b794570a1afa4771eff47dd39b43ec53a00c8b26dda95"} Jul 21 12:21:37.798578 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:37.798543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" event={"ID":"4506e84a-8330-468d-a823-549b20f8cd0d","Type":"ContainerStarted","Data":"9bb3efad09c80a87dd44d54101ca1bd11c47073e5f7cd9c53609d5e26262bd40"} Jul 21 12:21:38.803671 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:38.803633 2559 generic.go:356] "Generic (PLEG): container finished" podID="4506e84a-8330-468d-a823-549b20f8cd0d" containerID="9bb3efad09c80a87dd44d54101ca1bd11c47073e5f7cd9c53609d5e26262bd40" exitCode=0 Jul 21 12:21:38.804103 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:38.803706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" event={"ID":"4506e84a-8330-468d-a823-549b20f8cd0d","Type":"ContainerDied","Data":"9bb3efad09c80a87dd44d54101ca1bd11c47073e5f7cd9c53609d5e26262bd40"} Jul 21 12:21:39.809548 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:39.809506 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" event={"ID":"4506e84a-8330-468d-a823-549b20f8cd0d","Type":"ContainerStarted","Data":"7ceaad8b8241601d287f1220719574e81897a51805ae968bd46e3f635a467d1b"} Jul 21 12:21:39.809931 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:39.809606 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:39.843873 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:39.843815 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" podStartSLOduration=2.082597945 podStartE2EDuration="6.843797074s" totalImagesPullingTime="3.717813985s" totalInitContainerRuntime="1.043385144s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:21:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:21:39.842604626 +0000 UTC m=+626.011955902" watchObservedRunningTime="2026-07-21 12:21:39.843797074 +0000 UTC m=+626.013148349" Jul 21 12:21:49.141863 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:49.141831 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:49.220891 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:49.220850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:50.814925 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:50.814897 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58f85c6b8d-5qdxn" Jul 21 12:21:51.029614 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.029583 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf"] Jul 21 12:21:51.039642 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.039617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.042148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.042123 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 12:21:51.042371 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.042355 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 12:21:51.042459 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.042443 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vcw84\"" Jul 21 12:21:51.042513 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.042464 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 12:21:51.042513 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.042478 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 12:21:51.042513 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.042498 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 12:21:51.048334 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.048311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf"] Jul 21 12:21:51.168628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.168538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/044ea10b-97f0-4968-8a6b-ddd21255288f-cert\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.168628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.168573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw629\" (UniqueName: \"kubernetes.io/projected/044ea10b-97f0-4968-8a6b-ddd21255288f-kube-api-access-dw629\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.168628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.168603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/044ea10b-97f0-4968-8a6b-ddd21255288f-metrics-cert\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.168848 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.168693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/044ea10b-97f0-4968-8a6b-ddd21255288f-manager-config\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.270060 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.270017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/044ea10b-97f0-4968-8a6b-ddd21255288f-manager-config\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.270060 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.270066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/044ea10b-97f0-4968-8a6b-ddd21255288f-cert\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.270275 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.270085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw629\" (UniqueName: \"kubernetes.io/projected/044ea10b-97f0-4968-8a6b-ddd21255288f-kube-api-access-dw629\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.270319 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.270274 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/044ea10b-97f0-4968-8a6b-ddd21255288f-metrics-cert\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.270754 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.270729 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/044ea10b-97f0-4968-8a6b-ddd21255288f-manager-config\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.272565 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.272544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/044ea10b-97f0-4968-8a6b-ddd21255288f-metrics-cert\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.272654 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.272588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/044ea10b-97f0-4968-8a6b-ddd21255288f-cert\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.278367 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.278345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw629\" (UniqueName: \"kubernetes.io/projected/044ea10b-97f0-4968-8a6b-ddd21255288f-kube-api-access-dw629\") pod \"lws-controller-manager-c65cd9d66-wzrkf\" (UID: \"044ea10b-97f0-4968-8a6b-ddd21255288f\") " pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.349692 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.349658 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:51.483930 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.483897 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf"] Jul 21 12:21:51.487276 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:21:51.487245 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044ea10b_97f0_4968_8a6b_ddd21255288f.slice/crio-7467af038801aa70733c3c6a3053e4258878c55f268715a45e2b96ef3ceed560 WatchSource:0}: Error finding container 7467af038801aa70733c3c6a3053e4258878c55f268715a45e2b96ef3ceed560: Status 404 returned error can't find the container with id 7467af038801aa70733c3c6a3053e4258878c55f268715a45e2b96ef3ceed560 Jul 21 12:21:51.854152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:51.854116 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" event={"ID":"044ea10b-97f0-4968-8a6b-ddd21255288f","Type":"ContainerStarted","Data":"7467af038801aa70733c3c6a3053e4258878c55f268715a45e2b96ef3ceed560"} Jul 21 12:21:53.026874 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.026839 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:53.217607 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.217576 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:53.763294 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.763260 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:53.929439 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.929401 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x"] Jul 21 12:21:53.933176 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.933151 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:53.942443 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.942385 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 12:21:53.942443 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.942416 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 12:21:53.942443 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.942416 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 12:21:53.943224 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.943208 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ds55w\"" Jul 21 12:21:53.948147 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.948130 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 12:21:53.972616 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:53.972574 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x"] Jul 21 12:21:54.090615 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.090582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qm58\" (UniqueName: \"kubernetes.io/projected/ae191bd0-0927-4f31-8a15-f3d9b7be4934-kube-api-access-9qm58\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.091038 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.090641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae191bd0-0927-4f31-8a15-f3d9b7be4934-tls-certs\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.091038 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.090683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae191bd0-0927-4f31-8a15-f3d9b7be4934-tmp\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.192136 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.192095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qm58\" (UniqueName: \"kubernetes.io/projected/ae191bd0-0927-4f31-8a15-f3d9b7be4934-kube-api-access-9qm58\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.192334 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.192207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae191bd0-0927-4f31-8a15-f3d9b7be4934-tls-certs\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.192334 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.192233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae191bd0-0927-4f31-8a15-f3d9b7be4934-tmp\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.194444 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.194423 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae191bd0-0927-4f31-8a15-f3d9b7be4934-tmp\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.194884 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.194864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae191bd0-0927-4f31-8a15-f3d9b7be4934-tls-certs\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.195421 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.195403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:54.219723 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.219698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qm58\" (UniqueName: \"kubernetes.io/projected/ae191bd0-0927-4f31-8a15-f3d9b7be4934-kube-api-access-9qm58\") pod \"kube-auth-proxy-6fc49b7cdf-cf48x\" (UID: \"ae191bd0-0927-4f31-8a15-f3d9b7be4934\") " pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.244688 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.244653 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" Jul 21 12:21:54.423024 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.422997 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:54.423127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.423098 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x"] Jul 21 12:21:54.424846 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:21:54.424822 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae191bd0_0927_4f31_8a15_f3d9b7be4934.slice/crio-7c73f0f9bbb084dd04072a37d9eeaa855d4f07b7927311b2edf567abe7d6d701 WatchSource:0}: Error finding container 7c73f0f9bbb084dd04072a37d9eeaa855d4f07b7927311b2edf567abe7d6d701: Status 404 returned error can't find the container with id 7c73f0f9bbb084dd04072a37d9eeaa855d4f07b7927311b2edf567abe7d6d701 Jul 21 12:21:54.575905 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.575828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:54.736343 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.736312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:54.869791 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.869704 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" event={"ID":"044ea10b-97f0-4968-8a6b-ddd21255288f","Type":"ContainerStarted","Data":"c1503b95102f33be6fe410f603d70bf008d631a3ce6a8b616d89479d33f272d5"} Jul 21 12:21:54.869970 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.869822 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:21:54.871098 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:54.871075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" event={"ID":"ae191bd0-0927-4f31-8a15-f3d9b7be4934","Type":"ContainerStarted","Data":"7c73f0f9bbb084dd04072a37d9eeaa855d4f07b7927311b2edf567abe7d6d701"} Jul 21 12:21:55.005032 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:55.004980 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" podStartSLOduration=1.506556166 podStartE2EDuration="4.00496354s" totalImagesPullingTime="2.498407374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:21:54.997410559 +0000 UTC m=+641.166761836" watchObservedRunningTime="2026-07-21 12:21:55.00496354 +0000 UTC m=+641.174314807" Jul 21 12:21:55.869698 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:55.869654 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:56.053439 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:56.053378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:56.381150 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:56.381105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:56.531778 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:56.531723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:57.819294 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:57.819261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:21:57.889652 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:57.889613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" event={"ID":"ae191bd0-0927-4f31-8a15-f3d9b7be4934","Type":"ContainerStarted","Data":"44bb4b24cf7aade1fe5cb7d78b46ed9af8de69ca748711d7b02d31ce14f41bc0"} Jul 21 12:21:57.908235 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:21:57.908187 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fc49b7cdf-cf48x" podStartSLOduration=2.0834409640000002 podStartE2EDuration="4.908170185s" totalImagesPullingTime="2.824729221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:21:57.90719014 +0000 UTC m=+644.076541414" watchObservedRunningTime="2026-07-21 12:21:57.908170185 +0000 UTC m=+644.077521461" Jul 21 12:22:05.881662 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:05.881627 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c65cd9d66-wzrkf" Jul 21 12:22:06.778174 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:06.778142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:06.920914 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:06.920886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:07.031596 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:07.031515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:07.378292 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:07.378193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:07.654470 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:07.654364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:08.203559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:08.203524 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:08.340356 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:08.340328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:08.486731 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:08.486703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:08.911559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:08.911486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:09.321858 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:09.321829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:09.711805 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:09.711726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:54.371632 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:54.371560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:22:55.528039 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:22:55.528010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:14.618226 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:14.618144 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:18.819061 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:18.819031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:19.787649 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:19.787604 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:19.887368 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:19.887334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:19.949069 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:19.949038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:20.499059 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:20.499024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:20.599101 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:20.599067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:21.620156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:21.620111 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:25.073481 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:25.073431 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:26.225752 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:26.225712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:26.540714 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:26.540620 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:26.887858 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:26.887774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:28.273139 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:28.273107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:30.837088 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:30.837041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:30.939148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:30.939108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:31.049471 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:31.049441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:31.860220 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:31.860186 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:31.961677 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:31.961646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:34.121645 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.121613 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9"] Jul 21 12:23:34.124384 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.124361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.126658 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.126639 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 12:23:34.127506 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.127488 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mslb2\"" Jul 21 12:23:34.127581 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.127506 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 12:23:34.127581 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.127491 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 12:23:34.127581 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.127492 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 12:23:34.132705 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.132681 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9"] Jul 21 12:23:34.135719 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.135692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:34.225438 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.225407 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f0cceaf-5da2-493c-8d4c-0099fa790c90-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.225620 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.225444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7w72\" (UniqueName: \"kubernetes.io/projected/2f0cceaf-5da2-493c-8d4c-0099fa790c90-kube-api-access-g7w72\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.225620 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.225485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f0cceaf-5da2-493c-8d4c-0099fa790c90-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.326299 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.326249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f0cceaf-5da2-493c-8d4c-0099fa790c90-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.326299 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.326310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7w72\" (UniqueName: \"kubernetes.io/projected/2f0cceaf-5da2-493c-8d4c-0099fa790c90-kube-api-access-g7w72\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.326529 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.326350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f0cceaf-5da2-493c-8d4c-0099fa790c90-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.326529 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:23:34.326470 2559 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 21 12:23:34.326529 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:23:34.326527 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0cceaf-5da2-493c-8d4c-0099fa790c90-plugin-serving-cert podName:2f0cceaf-5da2-493c-8d4c-0099fa790c90 nodeName:}" failed. No retries permitted until 2026-07-21 12:23:34.826511959 +0000 UTC m=+740.995863212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2f0cceaf-5da2-493c-8d4c-0099fa790c90-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-8mtr9" (UID: "2f0cceaf-5da2-493c-8d4c-0099fa790c90") : secret "plugin-serving-cert" not found Jul 21 12:23:34.326921 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.326903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f0cceaf-5da2-493c-8d4c-0099fa790c90-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.335321 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.335296 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7w72\" (UniqueName: \"kubernetes.io/projected/2f0cceaf-5da2-493c-8d4c-0099fa790c90-kube-api-access-g7w72\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.650865 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.650829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:34.830720 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.830685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f0cceaf-5da2-493c-8d4c-0099fa790c90-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:34.833119 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:34.833095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f0cceaf-5da2-493c-8d4c-0099fa790c90-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mtr9\" (UID: \"2f0cceaf-5da2-493c-8d4c-0099fa790c90\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:35.033755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:35.033717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" Jul 21 12:23:35.160335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:35.160201 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9"] Jul 21 12:23:35.162926 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:23:35.162893 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f0cceaf_5da2_493c_8d4c_0099fa790c90.slice/crio-fe44f7648d7678b159ab4d7ea7306b4e57ec916a54c88d59c50abc7c7ae6ad20 WatchSource:0}: Error finding container fe44f7648d7678b159ab4d7ea7306b4e57ec916a54c88d59c50abc7c7ae6ad20: Status 404 returned error can't find the container with id fe44f7648d7678b159ab4d7ea7306b4e57ec916a54c88d59c50abc7c7ae6ad20 Jul 21 12:23:35.265311 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:35.265280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" event={"ID":"2f0cceaf-5da2-493c-8d4c-0099fa790c90","Type":"ContainerStarted","Data":"fe44f7648d7678b159ab4d7ea7306b4e57ec916a54c88d59c50abc7c7ae6ad20"} Jul 21 12:23:35.626261 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:35.626212 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:35.730966 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:35.730916 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:38.820529 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:38.820485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:38.921305 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:38.921266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:44.224925 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.224847 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq"] Jul 21 12:23:44.235101 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.235072 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:44.237852 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.237823 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ns8wt\"" Jul 21 12:23:44.241710 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.241685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq"] Jul 21 12:23:44.323918 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.323878 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nz4xq\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:44.323918 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.323919 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27b77\" (UniqueName: \"kubernetes.io/projected/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-kube-api-access-27b77\") pod \"kuadrant-operator-controller-manager-86878c769f-nz4xq\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:44.425364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.425332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nz4xq\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:44.425364 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.425374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27b77\" (UniqueName: \"kubernetes.io/projected/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-kube-api-access-27b77\") pod \"kuadrant-operator-controller-manager-86878c769f-nz4xq\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:44.425769 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.425748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nz4xq\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:44.438539 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.438487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27b77\" (UniqueName: \"kubernetes.io/projected/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-kube-api-access-27b77\") pod \"kuadrant-operator-controller-manager-86878c769f-nz4xq\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:44.550006 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:44.549905 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:23:47.091324 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:47.091289 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:47.188775 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:47.188745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:49.575519 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:49.575484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:49.677227 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:49.677099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:52.401040 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:52.401010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:52.501626 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:52.501591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:55.943481 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:55.943432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:56.045486 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:56.045454 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:23:57.423116 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:57.423090 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq"] Jul 21 12:23:57.425287 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:23:57.425263 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23dc475_ec5f_4e3f_8fd9_91ce2597a74f.slice/crio-c84bee2ce094feddd3dd8cedc8362c80545d977af214a5a4535f00663ac5c9df WatchSource:0}: Error finding container c84bee2ce094feddd3dd8cedc8362c80545d977af214a5a4535f00663ac5c9df: Status 404 returned error can't find the container with id c84bee2ce094feddd3dd8cedc8362c80545d977af214a5a4535f00663ac5c9df Jul 21 12:23:58.377714 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:58.377673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" event={"ID":"2f0cceaf-5da2-493c-8d4c-0099fa790c90","Type":"ContainerStarted","Data":"48d5d697caa4bd299658aaa76f71a442186c8df55c7a440b200ab25c4024901b"} Jul 21 12:23:58.378987 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:23:58.378943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" event={"ID":"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f","Type":"ContainerStarted","Data":"c84bee2ce094feddd3dd8cedc8362c80545d977af214a5a4535f00663ac5c9df"} Jul 21 12:24:03.401309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:03.401277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" event={"ID":"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f","Type":"ContainerStarted","Data":"46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2"} Jul 21 12:24:03.401728 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:03.401359 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:24:03.425241 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:03.425198 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mtr9" podStartSLOduration=7.225003329 podStartE2EDuration="29.425180417s" totalImagesPullingTime="22.200177088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:23:58.393093665 +0000 UTC m=+764.562444945" watchObservedRunningTime="2026-07-21 12:24:03.425180417 +0000 UTC m=+769.594531693" Jul 21 12:24:03.425370 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:03.425271 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" podStartSLOduration=14.291481717 podStartE2EDuration="19.42526653s" totalImagesPullingTime="5.133784813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:24:03.422847179 +0000 UTC m=+769.592198480" watchObservedRunningTime="2026-07-21 12:24:03.42526653 +0000 UTC m=+769.594617806" Jul 21 12:24:04.768274 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:04.768240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:04.869463 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:04.869434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:07.383107 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:07.383069 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:07.484015 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:07.483973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:10.588042 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:10.588005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:10.687909 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:10.687879 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:13.998637 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:13.998605 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:14.123212 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:14.123178 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:14.408619 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:14.408541 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:24:18.699473 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:18.699437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:18.761684 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:18.761647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:21.748804 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:21.748769 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:22.597641 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:22.597612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:22.696943 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:22.696910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:30.753235 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:30.753199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:31.114853 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:31.114775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:32.889479 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:32.889444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:32.998162 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:32.998122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:33.075545 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:33.075518 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:33.715869 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:33.715836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:33.775548 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:33.775522 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:33.867165 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:33.867133 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-ps86m"] Jul 21 12:24:34.164847 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.164810 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ps86m"] Jul 21 12:24:34.165270 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.164903 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ps86m" Jul 21 12:24:34.167304 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.167283 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jrdqk\"" Jul 21 12:24:34.259037 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.259004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnc7\" (UniqueName: \"kubernetes.io/projected/c811d0a8-ded7-41ae-a3c7-6a9bc9310d43-kube-api-access-mqnc7\") pod \"authorino-75d556bd77-ps86m\" (UID: \"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43\") " pod="kuadrant-system/authorino-75d556bd77-ps86m" Jul 21 12:24:34.360309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.360266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnc7\" (UniqueName: \"kubernetes.io/projected/c811d0a8-ded7-41ae-a3c7-6a9bc9310d43-kube-api-access-mqnc7\") pod \"authorino-75d556bd77-ps86m\" (UID: \"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43\") " pod="kuadrant-system/authorino-75d556bd77-ps86m" Jul 21 12:24:34.368302 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.368272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnc7\" (UniqueName: \"kubernetes.io/projected/c811d0a8-ded7-41ae-a3c7-6a9bc9310d43-kube-api-access-mqnc7\") pod \"authorino-75d556bd77-ps86m\" (UID: \"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43\") " pod="kuadrant-system/authorino-75d556bd77-ps86m" Jul 21 12:24:34.474222 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.474193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ps86m" Jul 21 12:24:34.599264 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:34.599236 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ps86m"] Jul 21 12:24:34.601463 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:24:34.601433 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc811d0a8_ded7_41ae_a3c7_6a9bc9310d43.slice/crio-378f5e1da5fc271f332f5969e10e203861dfc9a5ecfd476380c5a06329b524f4 WatchSource:0}: Error finding container 378f5e1da5fc271f332f5969e10e203861dfc9a5ecfd476380c5a06329b524f4: Status 404 returned error can't find the container with id 378f5e1da5fc271f332f5969e10e203861dfc9a5ecfd476380c5a06329b524f4 Jul 21 12:24:35.531638 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:35.531604 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ps86m" event={"ID":"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43","Type":"ContainerStarted","Data":"378f5e1da5fc271f332f5969e10e203861dfc9a5ecfd476380c5a06329b524f4"} Jul 21 12:24:37.547278 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:37.547243 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ps86m" event={"ID":"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43","Type":"ContainerStarted","Data":"1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe"} Jul 21 12:24:37.564027 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:37.563972 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-ps86m" podStartSLOduration=1.774346996 podStartE2EDuration="4.56394187s" totalImagesPullingTime="2.789594874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:24:37.561841922 +0000 UTC m=+803.731193198" watchObservedRunningTime="2026-07-21 12:24:37.56394187 +0000 UTC m=+803.733293146" Jul 21 12:24:48.986408 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:48.986375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:49.060586 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:49.060546 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:24:49.486678 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:24:49.486648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:01.420977 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:01.420914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:01.746115 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:01.746082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:03.164703 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.164669 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ps86m"] Jul 21 12:25:03.165213 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.164893 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-ps86m" podUID="c811d0a8-ded7-41ae-a3c7-6a9bc9310d43" containerName="authorino" containerID="cri-o://1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe" gracePeriod=30 Jul 21 12:25:03.417062 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.417003 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ps86m" Jul 21 12:25:03.493251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.493219 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqnc7\" (UniqueName: \"kubernetes.io/projected/c811d0a8-ded7-41ae-a3c7-6a9bc9310d43-kube-api-access-mqnc7\") pod \"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43\" (UID: \"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43\") " Jul 21 12:25:03.495421 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.495387 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c811d0a8-ded7-41ae-a3c7-6a9bc9310d43-kube-api-access-mqnc7" (OuterVolumeSpecName: "kube-api-access-mqnc7") pod "c811d0a8-ded7-41ae-a3c7-6a9bc9310d43" (UID: "c811d0a8-ded7-41ae-a3c7-6a9bc9310d43"). InnerVolumeSpecName "kube-api-access-mqnc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:25:03.594840 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.594808 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqnc7\" (UniqueName: \"kubernetes.io/projected/c811d0a8-ded7-41ae-a3c7-6a9bc9310d43-kube-api-access-mqnc7\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:25:03.661178 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.661148 2559 generic.go:356] "Generic (PLEG): container finished" podID="c811d0a8-ded7-41ae-a3c7-6a9bc9310d43" containerID="1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe" exitCode=0 Jul 21 12:25:03.661335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.661192 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ps86m" Jul 21 12:25:03.661335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.661197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ps86m" event={"ID":"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43","Type":"ContainerDied","Data":"1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe"} Jul 21 12:25:03.661335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.661228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ps86m" event={"ID":"c811d0a8-ded7-41ae-a3c7-6a9bc9310d43","Type":"ContainerDied","Data":"378f5e1da5fc271f332f5969e10e203861dfc9a5ecfd476380c5a06329b524f4"} Jul 21 12:25:03.661335 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.661246 2559 scope.go:117] "RemoveContainer" containerID="1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe" Jul 21 12:25:03.670263 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.670247 2559 scope.go:117] "RemoveContainer" containerID="1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe" Jul 21 12:25:03.670515 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:25:03.670499 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe\": container with ID starting with 1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe not found: ID does not exist" containerID="1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe" Jul 21 12:25:03.670567 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.670522 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe"} err="failed to get container status \"1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe\": rpc error: code = NotFound desc = could not find container \"1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe\": container with ID starting with 1a05b7c2fc9107daca855021278fb52f460a077e61489cb69f1e69e250aa57fe not found: ID does not exist" Jul 21 12:25:03.681937 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.681914 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ps86m"] Jul 21 12:25:03.683220 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:03.683196 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ps86m"] Jul 21 12:25:04.358265 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:04.358237 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c811d0a8-ded7-41ae-a3c7-6a9bc9310d43" path="/var/lib/kubelet/pods/c811d0a8-ded7-41ae-a3c7-6a9bc9310d43/volumes" Jul 21 12:25:07.832914 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:07.832882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:07.933257 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:07.933225 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:08.001124 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:08.001094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:10.988118 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:10.988075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:11.084888 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:11.084850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:36.098072 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:36.098038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:36.199038 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:36.199009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:42.598079 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:42.598044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:42.695563 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:42.695515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:43.129200 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.129169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:43.297674 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.297642 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:43.634654 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.634621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:43.738560 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.738524 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:43.821175 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.821144 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-p299v"] Jul 21 12:25:43.821485 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.821471 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c811d0a8-ded7-41ae-a3c7-6a9bc9310d43" containerName="authorino" Jul 21 12:25:43.821538 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.821486 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c811d0a8-ded7-41ae-a3c7-6a9bc9310d43" containerName="authorino" Jul 21 12:25:43.821585 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.821550 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c811d0a8-ded7-41ae-a3c7-6a9bc9310d43" containerName="authorino" Jul 21 12:25:43.823410 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.823390 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:43.825615 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.825594 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 12:25:43.825740 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.825704 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nx4lk\"" Jul 21 12:25:43.834520 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.834492 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-p299v"] Jul 21 12:25:43.852105 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.852079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5sk\" (UniqueName: \"kubernetes.io/projected/0bb69899-55f7-4121-a3e1-d6cef741bdc1-kube-api-access-zm5sk\") pod \"payload-pre-processing-7677b774bb-p299v\" (UID: \"0bb69899-55f7-4121-a3e1-d6cef741bdc1\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:43.852222 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.852108 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb69899-55f7-4121-a3e1-d6cef741bdc1-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-p299v\" (UID: \"0bb69899-55f7-4121-a3e1-d6cef741bdc1\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:43.952722 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.952650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5sk\" (UniqueName: \"kubernetes.io/projected/0bb69899-55f7-4121-a3e1-d6cef741bdc1-kube-api-access-zm5sk\") pod \"payload-pre-processing-7677b774bb-p299v\" (UID: \"0bb69899-55f7-4121-a3e1-d6cef741bdc1\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:43.952722 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.952687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb69899-55f7-4121-a3e1-d6cef741bdc1-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-p299v\" (UID: \"0bb69899-55f7-4121-a3e1-d6cef741bdc1\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:43.953445 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.953427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb69899-55f7-4121-a3e1-d6cef741bdc1-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-p299v\" (UID: \"0bb69899-55f7-4121-a3e1-d6cef741bdc1\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:43.961127 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:43.961099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5sk\" (UniqueName: \"kubernetes.io/projected/0bb69899-55f7-4121-a3e1-d6cef741bdc1-kube-api-access-zm5sk\") pod \"payload-pre-processing-7677b774bb-p299v\" (UID: \"0bb69899-55f7-4121-a3e1-d6cef741bdc1\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:44.133400 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:44.133360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:44.263128 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:44.263105 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-p299v"] Jul 21 12:25:44.265062 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:25:44.265031 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb69899_55f7_4121_a3e1_d6cef741bdc1.slice/crio-6348d9d38e682d7744eb396b33fd53af8f6661bbfe7b91acb3aa54cfeff3673a WatchSource:0}: Error finding container 6348d9d38e682d7744eb396b33fd53af8f6661bbfe7b91acb3aa54cfeff3673a: Status 404 returned error can't find the container with id 6348d9d38e682d7744eb396b33fd53af8f6661bbfe7b91acb3aa54cfeff3673a Jul 21 12:25:44.829860 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:44.829822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" event={"ID":"0bb69899-55f7-4121-a3e1-d6cef741bdc1","Type":"ContainerStarted","Data":"6348d9d38e682d7744eb396b33fd53af8f6661bbfe7b91acb3aa54cfeff3673a"} Jul 21 12:25:46.841267 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:46.841227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" event={"ID":"0bb69899-55f7-4121-a3e1-d6cef741bdc1","Type":"ContainerStarted","Data":"80012249b7c7fdf8912d3ab0a4cbad0901bd34dff3c66475c0ba3eedace3c392"} Jul 21 12:25:46.841712 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:46.841308 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:25:46.857853 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:46.857803 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" podStartSLOduration=1.702255493 podStartE2EDuration="3.857786604s" totalImagesPullingTime="2.155531111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:25:46.857100778 +0000 UTC m=+873.026452053" watchObservedRunningTime="2026-07-21 12:25:46.857786604 +0000 UTC m=+873.027137882" Jul 21 12:25:53.950081 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:53.950042 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:54.049326 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:54.049290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:25:57.847694 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:25:57.847663 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-p299v" Jul 21 12:26:07.309746 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:07.309651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:26:08.194055 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:08.194023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:26:12.270047 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.270014 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv"] Jul 21 12:26:12.274962 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.274915 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.277337 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.277316 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 12:26:12.277602 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.277587 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 12:26:12.278296 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.278279 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-46ngs\"" Jul 21 12:26:12.278399 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.278323 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 12:26:12.282504 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.282462 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv"] Jul 21 12:26:12.305116 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.305072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:26:12.393110 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.393078 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w8p\" (UniqueName: \"kubernetes.io/projected/1d6d1621-fdf2-4e70-8138-d335eeedb12b-kube-api-access-r6w8p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.393279 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.393125 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.393279 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.393159 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d1621-fdf2-4e70-8138-d335eeedb12b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.393279 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.393220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.393279 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.393244 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.393446 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.393280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494355 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494365 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494473 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w8p\" (UniqueName: \"kubernetes.io/projected/1d6d1621-fdf2-4e70-8138-d335eeedb12b-kube-api-access-r6w8p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494759 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d1621-fdf2-4e70-8138-d335eeedb12b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494759 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494746 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.494841 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.494811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.495027 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.495008 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.496692 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.496674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d1621-fdf2-4e70-8138-d335eeedb12b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.497150 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.497131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d1621-fdf2-4e70-8138-d335eeedb12b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.502185 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.502165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w8p\" (UniqueName: \"kubernetes.io/projected/1d6d1621-fdf2-4e70-8138-d335eeedb12b-kube-api-access-r6w8p\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n95kv\" (UID: \"1d6d1621-fdf2-4e70-8138-d335eeedb12b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.586306 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.586229 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:12.714278 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.714241 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv"] Jul 21 12:26:12.718998 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:26:12.718971 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6d1621_fdf2_4e70_8138_d335eeedb12b.slice/crio-ecaebf489cdd03fdbcb0bc557ab195400c2d6caaef062b675bf41bc0957df9c5 WatchSource:0}: Error finding container ecaebf489cdd03fdbcb0bc557ab195400c2d6caaef062b675bf41bc0957df9c5: Status 404 returned error can't find the container with id ecaebf489cdd03fdbcb0bc557ab195400c2d6caaef062b675bf41bc0957df9c5 Jul 21 12:26:12.950059 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:12.949981 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" event={"ID":"1d6d1621-fdf2-4e70-8138-d335eeedb12b","Type":"ContainerStarted","Data":"ecaebf489cdd03fdbcb0bc557ab195400c2d6caaef062b675bf41bc0957df9c5"} Jul 21 12:26:13.956570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:13.956536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" event={"ID":"1d6d1621-fdf2-4e70-8138-d335eeedb12b","Type":"ContainerStarted","Data":"6497f48bdf6c406753c025996973779ecb449185ebce396ff92a0df35ec79573"} Jul 21 12:26:13.975278 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:13.975234 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" podStartSLOduration=0.886881527 podStartE2EDuration="1.975217097s" totalImagesPullingTime="1.08833557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:26:13.973394867 +0000 UTC m=+900.142746144" watchObservedRunningTime="2026-07-21 12:26:13.975217097 +0000 UTC m=+900.144568373" Jul 21 12:26:14.274591 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:14.274562 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:26:14.274885 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:14.274869 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:26:22.586750 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:22.586723 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:22.587170 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:22.586770 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:22.599394 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:22.599373 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:23.010855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:23.010824 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n95kv" Jul 21 12:26:32.682848 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.682812 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn"] Jul 21 12:26:32.686611 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.686588 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.689175 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.689151 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 21 12:26:32.704984 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.704960 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn"] Jul 21 12:26:32.773834 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.773802 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tzn\" (UniqueName: \"kubernetes.io/projected/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-kube-api-access-m9tzn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.774015 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.773843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.774015 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.773868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.774015 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.773961 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.774015 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.773992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.774152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.774021 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.792356 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.792325 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:26:32.875032 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875032 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875032 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875245 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875245 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tzn\" (UniqueName: \"kubernetes.io/projected/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-kube-api-access-m9tzn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875245 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875245 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875478 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875589 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.875691 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.875667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.877447 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.877428 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.877638 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.877622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.882872 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.882848 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tzn\" (UniqueName: \"kubernetes.io/projected/f7814600-3a73-45d7-b8f2-dc48b3e81fe0-kube-api-access-m9tzn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-krgfn\" (UID: \"f7814600-3a73-45d7-b8f2-dc48b3e81fe0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:32.996584 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:32.996534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:33.125990 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:33.125944 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn"] Jul 21 12:26:33.127823 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:26:33.127797 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7814600_3a73_45d7_b8f2_dc48b3e81fe0.slice/crio-12d1fb87d33aa716b0b18849d3c65808deea709db87fd28fbd7236f815e65fdc WatchSource:0}: Error finding container 12d1fb87d33aa716b0b18849d3c65808deea709db87fd28fbd7236f815e65fdc: Status 404 returned error can't find the container with id 12d1fb87d33aa716b0b18849d3c65808deea709db87fd28fbd7236f815e65fdc Jul 21 12:26:34.048050 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:34.048018 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" event={"ID":"f7814600-3a73-45d7-b8f2-dc48b3e81fe0","Type":"ContainerStarted","Data":"a792a46c00dfc661989bd9276f502bbc478715a9c770decbcef0c1c7de9f4047"} Jul 21 12:26:34.048491 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:34.048066 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" event={"ID":"f7814600-3a73-45d7-b8f2-dc48b3e81fe0","Type":"ContainerStarted","Data":"12d1fb87d33aa716b0b18849d3c65808deea709db87fd28fbd7236f815e65fdc"} Jul 21 12:26:34.072939 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:34.072843 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" podStartSLOduration=1.773515054 podStartE2EDuration="2.072823734s" totalImagesPullingTime="299.30868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:26:34.072119442 +0000 UTC m=+920.241470717" watchObservedRunningTime="2026-07-21 12:26:34.072823734 +0000 UTC m=+920.242175014" Jul 21 12:26:42.997079 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:42.997023 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:42.997592 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:42.997103 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:43.009539 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:43.009512 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:43.096516 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:43.096483 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-krgfn" Jul 21 12:26:48.522488 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:48.522446 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:26:52.512299 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:52.512263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:26:58.877915 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:58.877885 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr"] Jul 21 12:26:58.881043 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:58.881026 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:58.883357 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:58.883338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 12:26:58.889377 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:58.889359 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr"] Jul 21 12:26:58.909821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:58.909796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:26:59.001124 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.001096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.001280 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.001155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.001280 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.001179 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.001280 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.001238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.001280 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.001255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hsxh\" (UniqueName: \"kubernetes.io/projected/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-kube-api-access-4hsxh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.001428 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.001295 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.101876 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.101836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.101889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.101981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.102007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hsxh\" (UniqueName: \"kubernetes.io/projected/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-kube-api-access-4hsxh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.102070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102304 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.102110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102386 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.102356 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102452 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.102432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.102502 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.102480 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.104152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.104134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.104304 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.104286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.110978 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.110939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hsxh\" (UniqueName: \"kubernetes.io/projected/82e35fbb-4aa3-4f8e-b791-b4b1668fe043-kube-api-access-4hsxh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr\" (UID: \"82e35fbb-4aa3-4f8e-b791-b4b1668fe043\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.191534 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.191460 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:26:59.322997 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.322976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr"] Jul 21 12:26:59.325016 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:26:59.324987 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e35fbb_4aa3_4f8e_b791_b4b1668fe043.slice/crio-3dfaa12b16425412ef7839618a09f5a764fafb662bd8292587d6187407d1a49b WatchSource:0}: Error finding container 3dfaa12b16425412ef7839618a09f5a764fafb662bd8292587d6187407d1a49b: Status 404 returned error can't find the container with id 3dfaa12b16425412ef7839618a09f5a764fafb662bd8292587d6187407d1a49b Jul 21 12:26:59.326278 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:26:59.326262 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 12:27:00.163687 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:00.163648 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" event={"ID":"82e35fbb-4aa3-4f8e-b791-b4b1668fe043","Type":"ContainerStarted","Data":"75db0d862a326e04621722996057777d26a7a267d9bb900dd642fb1d4bddb931"} Jul 21 12:27:00.164116 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:00.163696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" event={"ID":"82e35fbb-4aa3-4f8e-b791-b4b1668fe043","Type":"ContainerStarted","Data":"3dfaa12b16425412ef7839618a09f5a764fafb662bd8292587d6187407d1a49b"} Jul 21 12:27:00.181780 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:00.181732 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" podStartSLOduration=1.948912388 podStartE2EDuration="2.181715145s" totalImagesPullingTime="232.802757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:27:00.180258343 +0000 UTC m=+946.349609618" watchObservedRunningTime="2026-07-21 12:27:00.181715145 +0000 UTC m=+946.351066451" Jul 21 12:27:09.192046 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:09.192012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:27:09.192536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:09.192064 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:27:09.205336 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:09.205312 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:27:09.216632 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:09.216605 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr" Jul 21 12:27:23.277667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:23.277621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:27:46.674730 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:46.674646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:27:46.762426 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:46.762398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:27:47.759336 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:47.759299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:27:47.858156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:47.858120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:27:48.900077 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:48.900036 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:27:48.995392 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:27:48.995362 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:29:40.366603 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:29:40.366525 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:29:43.319583 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:29:43.319544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:29:43.735715 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:29:43.735686 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:29:59.534638 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:29:59.534597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:00.753172 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:00.753142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:02.077534 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:02.077497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:40.409197 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:40.409091 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:40.507023 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:40.506986 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:40.641991 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:40.641923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:46.306231 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:46.306200 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:46.357900 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:46.357852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:46.498202 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:46.498171 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:46.629825 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:46.629745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:46.723177 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:46.723146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:52.384998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:52.384940 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:30:52.450941 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:30:52.450911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:31:14.304054 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:31:14.304028 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:31:14.304533 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:31:14.304189 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:32:54.318112 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:32:54.318076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:32:54.402107 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:32:54.402075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:33:20.122792 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:33:20.122758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:34:44.810754 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:44.810720 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq"] Jul 21 12:34:44.811259 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:44.810921 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" podUID="b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" containerName="manager" containerID="cri-o://46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2" gracePeriod=10 Jul 21 12:34:48.659945 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:48.659918 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:34:48.802414 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:48.802380 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-extensions-socket-volume\") pod \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " Jul 21 12:34:48.802600 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:48.802508 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27b77\" (UniqueName: \"kubernetes.io/projected/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-kube-api-access-27b77\") pod \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\" (UID: \"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f\") " Jul 21 12:34:48.802807 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:48.802783 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" (UID: "b23dc475-ec5f-4e3f-8fd9-91ce2597a74f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:34:48.804455 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:48.804430 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-kube-api-access-27b77" (OuterVolumeSpecName: "kube-api-access-27b77") pod "b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" (UID: "b23dc475-ec5f-4e3f-8fd9-91ce2597a74f"). InnerVolumeSpecName "kube-api-access-27b77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:34:48.904185 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:48.904150 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27b77\" (UniqueName: \"kubernetes.io/projected/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-kube-api-access-27b77\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:34:48.904185 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:48.904181 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f-extensions-socket-volume\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:34:49.214142 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.214052 2559 generic.go:356] "Generic (PLEG): container finished" podID="b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" containerID="46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2" exitCode=0 Jul 21 12:34:49.214142 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.214095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" event={"ID":"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f","Type":"ContainerDied","Data":"46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2"} Jul 21 12:34:49.214142 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.214121 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" Jul 21 12:34:49.214362 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.214183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq" event={"ID":"b23dc475-ec5f-4e3f-8fd9-91ce2597a74f","Type":"ContainerDied","Data":"c84bee2ce094feddd3dd8cedc8362c80545d977af214a5a4535f00663ac5c9df"} Jul 21 12:34:49.214362 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.214214 2559 scope.go:117] "RemoveContainer" containerID="46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2" Jul 21 12:34:49.223851 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.223831 2559 scope.go:117] "RemoveContainer" containerID="46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2" Jul 21 12:34:49.224133 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:34:49.224111 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2\": container with ID starting with 46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2 not found: ID does not exist" containerID="46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2" Jul 21 12:34:49.224195 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.224141 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2"} err="failed to get container status \"46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2\": rpc error: code = NotFound desc = could not find container \"46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2\": container with ID starting with 46fb9b57511c472cd98a31404ba191c47f3fac6def85885aa01ee0fe174da5e2 not found: ID does not exist" Jul 21 12:34:49.236135 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.236110 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq"] Jul 21 12:34:49.243615 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:49.243592 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz4xq"] Jul 21 12:34:50.358769 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:34:50.358736 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" path="/var/lib/kubelet/pods/b23dc475-ec5f-4e3f-8fd9-91ce2597a74f/volumes" Jul 21 12:35:45.910303 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.910270 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t"] Jul 21 12:35:45.910815 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.910603 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" containerName="manager" Jul 21 12:35:45.910815 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.910615 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" containerName="manager" Jul 21 12:35:45.910815 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.910672 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b23dc475-ec5f-4e3f-8fd9-91ce2597a74f" containerName="manager" Jul 21 12:35:45.913503 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.913487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:45.916101 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.916084 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ns8wt\"" Jul 21 12:35:45.925424 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.925382 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t"] Jul 21 12:35:45.996369 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.996338 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226f6c60-cdce-48a7-8b29-ea86356b7b54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-khz5t\" (UID: \"226f6c60-cdce-48a7-8b29-ea86356b7b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:45.996558 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:45.996396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzlc\" (UniqueName: \"kubernetes.io/projected/226f6c60-cdce-48a7-8b29-ea86356b7b54-kube-api-access-2nzlc\") pod \"kuadrant-operator-controller-manager-86878c769f-khz5t\" (UID: \"226f6c60-cdce-48a7-8b29-ea86356b7b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:46.097473 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.097436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzlc\" (UniqueName: \"kubernetes.io/projected/226f6c60-cdce-48a7-8b29-ea86356b7b54-kube-api-access-2nzlc\") pod \"kuadrant-operator-controller-manager-86878c769f-khz5t\" (UID: \"226f6c60-cdce-48a7-8b29-ea86356b7b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:46.097641 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.097545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226f6c60-cdce-48a7-8b29-ea86356b7b54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-khz5t\" (UID: \"226f6c60-cdce-48a7-8b29-ea86356b7b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:46.097945 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.097927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226f6c60-cdce-48a7-8b29-ea86356b7b54-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-khz5t\" (UID: \"226f6c60-cdce-48a7-8b29-ea86356b7b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:46.105717 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.105697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzlc\" (UniqueName: \"kubernetes.io/projected/226f6c60-cdce-48a7-8b29-ea86356b7b54-kube-api-access-2nzlc\") pod \"kuadrant-operator-controller-manager-86878c769f-khz5t\" (UID: \"226f6c60-cdce-48a7-8b29-ea86356b7b54\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:46.223365 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.223332 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:46.370764 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.370734 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t"] Jul 21 12:35:46.374624 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:35:46.374602 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226f6c60_cdce_48a7_8b29_ea86356b7b54.slice/crio-1d6fa69ce85550be1c0dd382cd7e95b8729ab6c327c17ace052593865d6fce6b WatchSource:0}: Error finding container 1d6fa69ce85550be1c0dd382cd7e95b8729ab6c327c17ace052593865d6fce6b: Status 404 returned error can't find the container with id 1d6fa69ce85550be1c0dd382cd7e95b8729ab6c327c17ace052593865d6fce6b Jul 21 12:35:46.450548 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.450513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" event={"ID":"226f6c60-cdce-48a7-8b29-ea86356b7b54","Type":"ContainerStarted","Data":"568dcae328cf3bbcdf597770cb50e65ad58132e49e7ca1742b8d71778e07eef7"} Jul 21 12:35:46.450664 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.450579 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:35:46.450664 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.450605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" event={"ID":"226f6c60-cdce-48a7-8b29-ea86356b7b54","Type":"ContainerStarted","Data":"1d6fa69ce85550be1c0dd382cd7e95b8729ab6c327c17ace052593865d6fce6b"} Jul 21 12:35:46.479988 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:46.479885 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" podStartSLOduration=1.479871142 podStartE2EDuration="1.479871142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:35:46.478035468 +0000 UTC m=+1472.647386747" watchObservedRunningTime="2026-07-21 12:35:46.479871142 +0000 UTC m=+1472.649222417" Jul 21 12:35:57.459420 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:35:57.459390 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khz5t" Jul 21 12:36:14.329460 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:36:14.329433 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:36:14.331474 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:36:14.331453 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:38:36.745658 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:36.745583 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:38:39.282867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.282837 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9"] Jul 21 12:38:39.286443 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.286420 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.289845 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.289823 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-794e69c5-kserve-self-signed-certs\"" Jul 21 12:38:39.297746 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.297724 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9"] Jul 21 12:38:39.320297 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.320261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:38:39.372286 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.372256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-model-cache\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.372444 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.372319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tls-certs\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.372444 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.372359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-home\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.372444 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.372380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-dshm\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.372540 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.372439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tmp-dir\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.372658 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.372634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcnv\" (UniqueName: \"kubernetes.io/projected/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-kube-api-access-mdcnv\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.473540 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.473504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcnv\" (UniqueName: \"kubernetes.io/projected/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-kube-api-access-mdcnv\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.473706 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.473547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-model-cache\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.473706 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.473593 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tls-certs\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.473706 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.473628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-home\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.473706 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.473649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-dshm\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.473706 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.473677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tmp-dir\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.474097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.474047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-model-cache\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.474097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.474060 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-home\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.474178 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.474096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tmp-dir\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.475931 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.475913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-dshm\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.476269 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.476253 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tls-certs\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.481602 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.481581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcnv\" (UniqueName: \"kubernetes.io/projected/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-kube-api-access-mdcnv\") pod \"e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.595902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.595819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:39.727881 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:39.727860 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9"] Jul 21 12:38:39.729702 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:38:39.729676 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef5c50c7_fb81_4c11_aca3_fccf8fae8897.slice/crio-5468fdc20f6bee07709b11452611a3e16e0233bf5113e47f4e767b6b19ca8a54 WatchSource:0}: Error finding container 5468fdc20f6bee07709b11452611a3e16e0233bf5113e47f4e767b6b19ca8a54: Status 404 returned error can't find the container with id 5468fdc20f6bee07709b11452611a3e16e0233bf5113e47f4e767b6b19ca8a54 Jul 21 12:38:40.211678 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:40.211642 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" event={"ID":"ef5c50c7-fb81-4c11-aca3-fccf8fae8897","Type":"ContainerStarted","Data":"c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776"} Jul 21 12:38:40.211678 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:40.211679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" event={"ID":"ef5c50c7-fb81-4c11-aca3-fccf8fae8897","Type":"ContainerStarted","Data":"5468fdc20f6bee07709b11452611a3e16e0233bf5113e47f4e767b6b19ca8a54"} Jul 21 12:38:40.236066 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:40.236009 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" podStartSLOduration=1.235988284 podStartE2EDuration="1.235988284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:38:40.233393297 +0000 UTC m=+1646.402744566" watchObservedRunningTime="2026-07-21 12:38:40.235988284 +0000 UTC m=+1646.405339560" Jul 21 12:38:42.330153 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:42.330116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:38:49.596902 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:49.596872 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:49.597327 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:49.596917 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:49.608969 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:49.608924 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:38:50.269239 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:38:50.269210 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:39:15.831303 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:15.831272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:39:16.560893 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:16.560857 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9"] Jul 21 12:39:16.577425 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:16.577398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:39:31.581132 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.581070 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" podUID="ef5c50c7-fb81-4c11-aca3-fccf8fae8897" containerName="main" containerID="cri-o://c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776" gracePeriod=45 Jul 21 12:39:31.825047 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.825016 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:39:31.944553 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944446 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-model-cache\") pod \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " Jul 21 12:39:31.944553 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944541 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdcnv\" (UniqueName: \"kubernetes.io/projected/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-kube-api-access-mdcnv\") pod \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " Jul 21 12:39:31.944855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944575 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tls-certs\") pod \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " Jul 21 12:39:31.944855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944606 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tmp-dir\") pod \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " Jul 21 12:39:31.944855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944680 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-dshm\") pod \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " Jul 21 12:39:31.944855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-home\") pod \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\" (UID: \"ef5c50c7-fb81-4c11-aca3-fccf8fae8897\") " Jul 21 12:39:31.944855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944711 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef5c50c7-fb81-4c11-aca3-fccf8fae8897" (UID: "ef5c50c7-fb81-4c11-aca3-fccf8fae8897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:39:31.945155 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.944935 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef5c50c7-fb81-4c11-aca3-fccf8fae8897" (UID: "ef5c50c7-fb81-4c11-aca3-fccf8fae8897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:39:31.945155 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.945068 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:39:31.945155 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.945085 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:39:31.945320 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.945230 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-home" (OuterVolumeSpecName: "home") pod "ef5c50c7-fb81-4c11-aca3-fccf8fae8897" (UID: "ef5c50c7-fb81-4c11-aca3-fccf8fae8897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:39:31.946916 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.946887 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef5c50c7-fb81-4c11-aca3-fccf8fae8897" (UID: "ef5c50c7-fb81-4c11-aca3-fccf8fae8897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:39:31.946916 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.946899 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-kube-api-access-mdcnv" (OuterVolumeSpecName: "kube-api-access-mdcnv") pod "ef5c50c7-fb81-4c11-aca3-fccf8fae8897" (UID: "ef5c50c7-fb81-4c11-aca3-fccf8fae8897"). InnerVolumeSpecName "kube-api-access-mdcnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:39:31.946916 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:31.946902 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-dshm" (OuterVolumeSpecName: "dshm") pod "ef5c50c7-fb81-4c11-aca3-fccf8fae8897" (UID: "ef5c50c7-fb81-4c11-aca3-fccf8fae8897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:39:32.046250 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.046216 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:39:32.046250 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.046259 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:39:32.046473 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.046272 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdcnv\" (UniqueName: \"kubernetes.io/projected/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-kube-api-access-mdcnv\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:39:32.046473 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.046287 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5c50c7-fb81-4c11-aca3-fccf8fae8897-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:39:32.441996 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.441932 2559 generic.go:356] "Generic (PLEG): container finished" podID="ef5c50c7-fb81-4c11-aca3-fccf8fae8897" containerID="c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776" exitCode=0 Jul 21 12:39:32.442191 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.442020 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" Jul 21 12:39:32.442191 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.442019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" event={"ID":"ef5c50c7-fb81-4c11-aca3-fccf8fae8897","Type":"ContainerDied","Data":"c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776"} Jul 21 12:39:32.442191 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.442071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9" event={"ID":"ef5c50c7-fb81-4c11-aca3-fccf8fae8897","Type":"ContainerDied","Data":"5468fdc20f6bee07709b11452611a3e16e0233bf5113e47f4e767b6b19ca8a54"} Jul 21 12:39:32.442191 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.442094 2559 scope.go:117] "RemoveContainer" containerID="c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776" Jul 21 12:39:32.450517 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.450501 2559 scope.go:117] "RemoveContainer" containerID="c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776" Jul 21 12:39:32.450750 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:39:32.450734 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776\": container with ID starting with c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776 not found: ID does not exist" containerID="c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776" Jul 21 12:39:32.450823 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.450756 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776"} err="failed to get container status \"c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776\": rpc error: code = NotFound desc = could not find container \"c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776\": container with ID starting with c3dbc4d4d63d0d469b8e077374791c10f4dde1a11b44214b33eee33f23421776 not found: ID does not exist" Jul 21 12:39:32.459663 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.459636 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9"] Jul 21 12:39:32.462932 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:32.462908 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-794e69c5-kserve-7545d5c695-fn7w9"] Jul 21 12:39:34.358272 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:39:34.358231 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5c50c7-fb81-4c11-aca3-fccf8fae8897" path="/var/lib/kubelet/pods/ef5c50c7-fb81-4c11-aca3-fccf8fae8897/volumes" Jul 21 12:41:05.537039 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:41:05.537001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:41:14.356194 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:41:14.356127 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:41:14.359930 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:41:14.359910 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:41:37.363425 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:41:37.363382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:41:37.420848 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:41:37.420815 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:21.957756 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:21.957716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:43.496206 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:43.496123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:46.404076 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:46.404042 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:46.503907 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:46.503870 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:46.860086 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:46.860052 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:52.736314 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:52.736276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:52.801825 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:52.801796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:52.945814 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:52.945778 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:53.254845 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:53.254812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:53.558691 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:53.558611 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:58.193342 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:58.193311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:58.284981 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:58.284935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:58.366565 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:58.366531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:58.528845 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:58.528810 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:58.907725 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:58.907640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:42:59.070937 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:42:59.070904 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:34.149105 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:34.149072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:38.781668 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:38.781629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:40.060107 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:40.060074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:40.160882 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:40.160845 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:42.530458 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:42.530423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:42.642135 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:42.642107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:42.880908 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:42.880813 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:43.232821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:43.232789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:43:43.334596 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:43:43.334551 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:18.093099 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:18.093012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:22.679243 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:22.679205 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:22.890443 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:22.890374 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:23.129457 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:23.129421 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:23.358000 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:23.357967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:23.742709 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:23.742678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:24.170417 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:24.170275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:25.701850 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:25.701816 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:25.818485 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:25.818455 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:25.906993 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:25.906939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:26.062862 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:26.062826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:26.483060 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:26.483021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:26.583907 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:26.583876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:44:37.363869 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:44:37.363834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:02.245826 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:02.245792 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:06.637273 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:06.637229 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:07.698359 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:07.698310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:07.945615 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:07.945584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:08.154632 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:08.154603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:08.487819 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:08.487785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:08.644784 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:08.644753 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:10.912063 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:10.912018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:11.003525 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:11.003493 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:11.090299 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:11.090257 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:11.761100 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:11.761072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:11.861470 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:11.861438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:12.118733 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:12.118617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:22.425291 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:22.425255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:22.567969 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:22.567928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:24.618840 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:24.618804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:33.471078 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:33.471045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:33.570442 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:33.570409 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:46.294805 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:46.294725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:50.898223 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:50.898187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:51.086582 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:51.086537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:51.336357 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:51.336322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:51.861368 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:51.861315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:51.952650 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:51.952617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:52.046738 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:52.046665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:53.553657 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:53.553610 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:59.358659 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:59.358627 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:59.426854 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:59.426821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:59.517501 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:59.517471 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:59.615248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:59.615157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:45:59.714830 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:45:59.714793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:00.145139 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:00.145100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:14.382508 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:14.382479 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:46:14.389960 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:14.389921 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:46:26.115662 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:26.115623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:32.537729 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:32.537696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:32.638675 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:32.638643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:32.734228 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:32.734193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:48.601418 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:48.601386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:54.245504 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:54.245466 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:54.314269 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:54.314230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:54.490729 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:54.490693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:46:54.601805 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:46:54.601721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:04.961684 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:04.961651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:05.747604 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:05.747526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:12.206240 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:12.206173 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:12.312098 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:12.312059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:12.406365 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:12.406315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:12.508002 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:12.507968 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:13.222824 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:13.222776 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:13.317704 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:13.317663 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:13.417385 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:13.417328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:14.931059 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:14.930981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:21.619322 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:21.619286 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:21.717431 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:21.717390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:22.329242 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:22.329211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:28.470326 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:28.470287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:28.564907 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:28.564862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:28.710796 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:28.710761 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:29.035026 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:29.034990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:29.492653 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:29.492617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:32.257636 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:32.257603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:32.356179 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:32.356151 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:32.559654 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:32.559570 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:32.766200 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:32.766169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:33.093481 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:33.093450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:33.196765 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:33.196736 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:36.633316 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:36.633277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:37.328591 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:37.328557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:37.423350 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:37.423302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:37.693463 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:37.693368 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:43.426309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:43.426273 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:43.489993 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:43.489924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:45.558047 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:45.558010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:54.236884 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:54.236841 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:58.861583 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:58.861544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:47:59.240731 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:47:59.240701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:04.898523 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:04.898486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:04.998585 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:04.998554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:05.155365 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:05.155280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:05.379612 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:05.379578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:05.488745 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:05.488716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:05.671274 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:05.671245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:05.768230 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:05.768152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:11.765528 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:11.765491 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:11.830738 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:11.830710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:11.916897 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:11.916853 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:12.018844 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:12.018759 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:12.123613 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:12.123579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:14.394830 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:14.394803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:14.553346 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:14.553318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:18.090059 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:18.090026 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:18.152575 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:18.152539 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:24.770360 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:24.770313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:25.063894 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:25.063808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:25.333483 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:25.333406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:28.543104 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:28.543071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:28.644091 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:28.644065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:28.737146 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:28.737112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:29.257810 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:29.257768 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:29.362464 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:29.362432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:32.916618 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:32.916581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:37.112437 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.112235 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj"] Jul 21 12:48:37.113263 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.113238 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5c50c7-fb81-4c11-aca3-fccf8fae8897" containerName="main" Jul 21 12:48:37.113334 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.113265 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5c50c7-fb81-4c11-aca3-fccf8fae8897" containerName="main" Jul 21 12:48:37.113381 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.113339 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5c50c7-fb81-4c11-aca3-fccf8fae8897" containerName="main" Jul 21 12:48:37.116465 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.116443 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.119741 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.119717 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ffb6e94e\"/\"e2e-lifecycle-model-ffb6e94e-kserve-self-signed-certs\"" Jul 21 12:48:37.120690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.120671 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ffb6e94e\"/\"kube-root-ca.crt\"" Jul 21 12:48:37.120810 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.120712 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ffb6e94e\"/\"openshift-service-ca.crt\"" Jul 21 12:48:37.120885 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.120851 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ffb6e94e\"/\"default-dockercfg-mktwn\"" Jul 21 12:48:37.132908 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.132880 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj"] Jul 21 12:48:37.165993 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.165937 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:48:37.283270 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.283233 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-home\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.283270 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.283270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-model-cache\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.283503 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.283299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-dshm\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.283503 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.283365 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394a6fb-4470-4762-b36a-60574c36c9ed-tls-certs\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.283503 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.283397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-tmp-dir\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.283503 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.283424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsr88\" (UniqueName: \"kubernetes.io/projected/2394a6fb-4470-4762-b36a-60574c36c9ed-kube-api-access-wsr88\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.384417 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.384334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-dshm\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.384417 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.384372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394a6fb-4470-4762-b36a-60574c36c9ed-tls-certs\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.384625 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.384496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-tmp-dir\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.384625 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.384550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsr88\" (UniqueName: \"kubernetes.io/projected/2394a6fb-4470-4762-b36a-60574c36c9ed-kube-api-access-wsr88\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.384730 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.384660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-home\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.384730 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.384688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-model-cache\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.385049 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.385019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-tmp-dir\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.385049 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.385036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-home\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.385253 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.385076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-model-cache\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.386607 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.386584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-dshm\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.386918 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.386901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394a6fb-4470-4762-b36a-60574c36c9ed-tls-certs\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.392023 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.392003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsr88\" (UniqueName: \"kubernetes.io/projected/2394a6fb-4470-4762-b36a-60574c36c9ed-kube-api-access-wsr88\") pod \"e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.429079 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.429050 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:37.567370 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.567345 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj"] Jul 21 12:48:37.569864 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:48:37.569835 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2394a6fb_4470_4762_b36a_60574c36c9ed.slice/crio-9a156cdd14a572121fa356bbb2a429785acc3bf3ac8bdddc6abdbd338d3e52cc WatchSource:0}: Error finding container 9a156cdd14a572121fa356bbb2a429785acc3bf3ac8bdddc6abdbd338d3e52cc: Status 404 returned error can't find the container with id 9a156cdd14a572121fa356bbb2a429785acc3bf3ac8bdddc6abdbd338d3e52cc Jul 21 12:48:37.820934 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.820895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" event={"ID":"2394a6fb-4470-4762-b36a-60574c36c9ed","Type":"ContainerStarted","Data":"add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975"} Jul 21 12:48:37.821152 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.820941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" event={"ID":"2394a6fb-4470-4762-b36a-60574c36c9ed","Type":"ContainerStarted","Data":"9a156cdd14a572121fa356bbb2a429785acc3bf3ac8bdddc6abdbd338d3e52cc"} Jul 21 12:48:37.846775 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:37.846724 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" podStartSLOduration=0.846706995 podStartE2EDuration="846.706995ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:48:37.845540157 +0000 UTC m=+2244.014891445" watchObservedRunningTime="2026-07-21 12:48:37.846706995 +0000 UTC m=+2244.016058312" Jul 21 12:48:47.430175 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:47.430143 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:47.430175 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:47.430185 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:47.447318 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:47.447290 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:48:47.876891 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:48:47.876866 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:49:16.772731 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:16.772699 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:49:21.163048 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:21.163013 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:49:23.398320 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:23.398291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:49:28.369688 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:28.369651 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj"] Jul 21 12:49:28.467565 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:28.467538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:49:43.389571 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.389530 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" podUID="2394a6fb-4470-4762-b36a-60574c36c9ed" containerName="main" containerID="cri-o://add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975" gracePeriod=45 Jul 21 12:49:43.630562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.630537 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:49:43.676251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676183 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394a6fb-4470-4762-b36a-60574c36c9ed-tls-certs\") pod \"2394a6fb-4470-4762-b36a-60574c36c9ed\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " Jul 21 12:49:43.676251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676217 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-model-cache\") pod \"2394a6fb-4470-4762-b36a-60574c36c9ed\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " Jul 21 12:49:43.676549 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676278 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsr88\" (UniqueName: \"kubernetes.io/projected/2394a6fb-4470-4762-b36a-60574c36c9ed-kube-api-access-wsr88\") pod \"2394a6fb-4470-4762-b36a-60574c36c9ed\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " Jul 21 12:49:43.676549 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676303 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-dshm\") pod \"2394a6fb-4470-4762-b36a-60574c36c9ed\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " Jul 21 12:49:43.676549 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676320 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-tmp-dir\") pod \"2394a6fb-4470-4762-b36a-60574c36c9ed\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " Jul 21 12:49:43.676549 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676365 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-home\") pod \"2394a6fb-4470-4762-b36a-60574c36c9ed\" (UID: \"2394a6fb-4470-4762-b36a-60574c36c9ed\") " Jul 21 12:49:43.676768 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676738 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-home" (OuterVolumeSpecName: "home") pod "2394a6fb-4470-4762-b36a-60574c36c9ed" (UID: "2394a6fb-4470-4762-b36a-60574c36c9ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:49:43.676850 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676827 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2394a6fb-4470-4762-b36a-60574c36c9ed" (UID: "2394a6fb-4470-4762-b36a-60574c36c9ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:49:43.676916 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.676834 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "2394a6fb-4470-4762-b36a-60574c36c9ed" (UID: "2394a6fb-4470-4762-b36a-60574c36c9ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:49:43.678476 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.678443 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2394a6fb-4470-4762-b36a-60574c36c9ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2394a6fb-4470-4762-b36a-60574c36c9ed" (UID: "2394a6fb-4470-4762-b36a-60574c36c9ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:49:43.678476 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.678453 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2394a6fb-4470-4762-b36a-60574c36c9ed-kube-api-access-wsr88" (OuterVolumeSpecName: "kube-api-access-wsr88") pod "2394a6fb-4470-4762-b36a-60574c36c9ed" (UID: "2394a6fb-4470-4762-b36a-60574c36c9ed"). InnerVolumeSpecName "kube-api-access-wsr88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:49:43.678647 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.678551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-dshm" (OuterVolumeSpecName: "dshm") pod "2394a6fb-4470-4762-b36a-60574c36c9ed" (UID: "2394a6fb-4470-4762-b36a-60574c36c9ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:49:43.777970 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.777923 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:49:43.777970 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.777970 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:49:43.778168 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.777979 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:49:43.778168 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.777987 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394a6fb-4470-4762-b36a-60574c36c9ed-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:49:43.778168 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.777996 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394a6fb-4470-4762-b36a-60574c36c9ed-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:49:43.778168 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:43.778005 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsr88\" (UniqueName: \"kubernetes.io/projected/2394a6fb-4470-4762-b36a-60574c36c9ed-kube-api-access-wsr88\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:49:44.114604 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.114567 2559 generic.go:356] "Generic (PLEG): container finished" podID="2394a6fb-4470-4762-b36a-60574c36c9ed" containerID="add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975" exitCode=0 Jul 21 12:49:44.114779 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.114611 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" Jul 21 12:49:44.114779 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.114655 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" event={"ID":"2394a6fb-4470-4762-b36a-60574c36c9ed","Type":"ContainerDied","Data":"add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975"} Jul 21 12:49:44.114779 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.114700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj" event={"ID":"2394a6fb-4470-4762-b36a-60574c36c9ed","Type":"ContainerDied","Data":"9a156cdd14a572121fa356bbb2a429785acc3bf3ac8bdddc6abdbd338d3e52cc"} Jul 21 12:49:44.114779 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.114723 2559 scope.go:117] "RemoveContainer" containerID="add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975" Jul 21 12:49:44.123861 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.123841 2559 scope.go:117] "RemoveContainer" containerID="add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975" Jul 21 12:49:44.124124 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:49:44.124107 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975\": container with ID starting with add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975 not found: ID does not exist" containerID="add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975" Jul 21 12:49:44.124181 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.124133 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975"} err="failed to get container status \"add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975\": rpc error: code = NotFound desc = could not find container \"add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975\": container with ID starting with add1f3ce2d635e1a6d3e4d18c84b8cb807542bc8c620f4c4d2e0330e103df975 not found: ID does not exist" Jul 21 12:49:44.144847 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.144822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj"] Jul 21 12:49:44.151096 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.151076 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ffb6e94e/e2e-lifecycle-model-ffb6e94e-kserve-7cf99d586c-2cwcj"] Jul 21 12:49:44.357903 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:49:44.357868 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2394a6fb-4470-4762-b36a-60574c36c9ed" path="/var/lib/kubelet/pods/2394a6fb-4470-4762-b36a-60574c36c9ed/volumes" Jul 21 12:50:04.671226 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:04.671190 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:05.666283 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:05.666209 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:11.300464 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:11.300422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:11.872157 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:11.872123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:11.970945 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:11.970911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:12.845920 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:12.845881 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:12.941184 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:12.941152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:13.047531 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:13.047501 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:13.768122 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:13.768079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:13.868914 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:13.868879 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:13.971935 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:13.971903 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:15.342292 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:15.342261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:21.200126 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:21.200078 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:21.261536 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:21.261503 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:21.872872 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:21.872829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:23.404993 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:23.404941 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:23.495560 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:23.495532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:28.464187 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:28.464153 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:28.567062 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:28.567030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:28.661303 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:28.661267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:39.197541 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:39.197510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:50:39.554275 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:50:39.554240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:00.493288 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:00.493255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:06.546580 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:06.546540 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:06.647156 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:06.647124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:06.893919 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:06.893836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:07.223262 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:07.223233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:07.792614 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:07.792581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:10.252867 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:10.252836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:10.351559 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:10.351525 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:10.425669 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:10.425632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:10.597155 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:10.597071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:10.917129 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:10.917045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:11.017647 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:11.017613 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:14.407786 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:14.407761 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:51:14.415748 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:14.415728 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:51:14.734220 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:14.734184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:15.061668 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:15.061588 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:15.381092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:15.381018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:15.651173 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:15.651091 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:18.302729 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:18.302699 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:18.392315 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:18.392287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:18.557090 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:18.557014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:18.731055 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:18.731020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:19.236673 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:19.236643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:19.337284 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:19.337250 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:20.022679 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:20.022644 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:22.790442 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:22.790376 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:31.817686 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:31.817651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:31.917457 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:31.917424 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:36.047129 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:36.047058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:40.292733 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.292703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:40.390829 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.390799 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj"] Jul 21 12:51:40.391160 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.391146 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2394a6fb-4470-4762-b36a-60574c36c9ed" containerName="main" Jul 21 12:51:40.391212 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.391162 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2394a6fb-4470-4762-b36a-60574c36c9ed" containerName="main" Jul 21 12:51:40.391251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.391223 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2394a6fb-4470-4762-b36a-60574c36c9ed" containerName="main" Jul 21 12:51:40.393241 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.393203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:40.394112 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.394098 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.397427 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.397402 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1cb55b\"/\"openshift-service-ca.crt\"" Jul 21 12:51:40.397427 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.397419 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1cb55b\"/\"kube-root-ca.crt\"" Jul 21 12:51:40.397427 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.397422 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1cb55b\"/\"test-model-1cb55b-kserve-self-signed-certs\"" Jul 21 12:51:40.397672 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.397407 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1cb55b\"/\"default-dockercfg-2jkrc\"" Jul 21 12:51:40.405192 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.405171 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj"] Jul 21 12:51:40.476276 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.476243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:51:40.501248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.501216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-dshm\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.501389 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.501276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tmp-dir\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.501389 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.501303 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tls-certs\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.501389 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.501352 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhgw\" (UniqueName: \"kubernetes.io/projected/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-kube-api-access-zbhgw\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.501492 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.501391 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-home\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.501679 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.501642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-model-cache\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.602701 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.602617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tmp-dir\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.602701 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.602648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tls-certs\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.602701 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.602680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhgw\" (UniqueName: \"kubernetes.io/projected/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-kube-api-access-zbhgw\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.603021 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.602795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-home\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.603021 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.602883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-model-cache\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.603021 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.602943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-dshm\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.603170 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.603088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tmp-dir\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.603170 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.603128 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-home\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.603399 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.603369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-model-cache\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.605092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.605073 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-dshm\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.605395 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.605374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tls-certs\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.610649 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.610631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhgw\" (UniqueName: \"kubernetes.io/projected/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-kube-api-access-zbhgw\") pod \"test-model-1cb55b-kserve-664cb8f98f-6mqvj\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.703307 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.703268 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:40.839314 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:40.839274 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj"] Jul 21 12:51:40.842869 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:51:40.842832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e7c5a84_af42_4a0a_aa84_88046bf5af8b.slice/crio-dca339c7eaf0f496ba68ba337179f8d812811faf7be620808cee8640a2305093 WatchSource:0}: Error finding container dca339c7eaf0f496ba68ba337179f8d812811faf7be620808cee8640a2305093: Status 404 returned error can't find the container with id dca339c7eaf0f496ba68ba337179f8d812811faf7be620808cee8640a2305093 Jul 21 12:51:41.639363 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:41.639326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" event={"ID":"1e7c5a84-af42-4a0a-aa84-88046bf5af8b","Type":"ContainerStarted","Data":"89f3d689f1202265e91dfc695f764fba15e310fc621f247327da20cc6b0b5957"} Jul 21 12:51:41.639878 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:41.639377 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" event={"ID":"1e7c5a84-af42-4a0a-aa84-88046bf5af8b","Type":"ContainerStarted","Data":"dca339c7eaf0f496ba68ba337179f8d812811faf7be620808cee8640a2305093"} Jul 21 12:51:41.657513 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:41.657460 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" podStartSLOduration=1.657443021 podStartE2EDuration="1.657443021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:51:41.65671725 +0000 UTC m=+2427.826068525" watchObservedRunningTime="2026-07-21 12:51:41.657443021 +0000 UTC m=+2427.826794320" Jul 21 12:51:50.703468 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:50.703438 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:50.703994 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:50.703483 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:50.715906 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:50.715880 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:51:51.693550 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:51:51.693519 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:52:22.987064 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:22.987023 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487"] Jul 21 12:52:22.990174 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:22.990156 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:22.993543 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:22.993519 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ded38a\"/\"default-dockercfg-h9qkh\"" Jul 21 12:52:22.993663 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:22.993543 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ded38a\"/\"kube-root-ca.crt\"" Jul 21 12:52:22.993663 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:22.993542 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ded38a\"/\"test-model-ded38a-kserve-self-signed-certs\"" Jul 21 12:52:22.993663 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:22.993519 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ded38a\"/\"openshift-service-ca.crt\"" Jul 21 12:52:23.005704 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.005684 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487"] Jul 21 12:52:23.017145 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.017123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:52:23.074244 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.074216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-model-cache\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.074403 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.074270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-home\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.074403 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.074384 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tmp-dir\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.074481 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.074424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-dshm\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.074531 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.074512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tls-certs\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.074571 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.074548 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshrq\" (UniqueName: \"kubernetes.io/projected/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-kube-api-access-rshrq\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.175895 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.175864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tmp-dir\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.175895 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.175899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-dshm\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.176117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.175924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tls-certs\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.176117 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.176061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshrq\" (UniqueName: \"kubernetes.io/projected/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-kube-api-access-rshrq\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.176227 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.176117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-model-cache\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.176227 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.176166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-home\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.176331 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.176292 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tmp-dir\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.176466 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.176445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-model-cache\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.176535 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.176520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-home\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.178348 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.178309 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-dshm\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.178456 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.178436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tls-certs\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.185736 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.185707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshrq\" (UniqueName: \"kubernetes.io/projected/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-kube-api-access-rshrq\") pod \"test-model-ded38a-kserve-68d45948b7-8k487\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.299838 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.299746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:23.433663 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.433631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487"] Jul 21 12:52:23.437002 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:52:23.436977 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67ed5331_ce9e_47d4_a64d_f1daf8de4a89.slice/crio-c090bdd3062ba04746822395821c4f63c09acb3272cdd687edf13a0108670774 WatchSource:0}: Error finding container c090bdd3062ba04746822395821c4f63c09acb3272cdd687edf13a0108670774: Status 404 returned error can't find the container with id c090bdd3062ba04746822395821c4f63c09acb3272cdd687edf13a0108670774 Jul 21 12:52:23.821426 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.821393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" event={"ID":"67ed5331-ce9e-47d4-a64d-f1daf8de4a89","Type":"ContainerStarted","Data":"2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0"} Jul 21 12:52:23.821426 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.821430 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" event={"ID":"67ed5331-ce9e-47d4-a64d-f1daf8de4a89","Type":"ContainerStarted","Data":"c090bdd3062ba04746822395821c4f63c09acb3272cdd687edf13a0108670774"} Jul 21 12:52:23.839823 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:23.839765 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" podStartSLOduration=1.839746578 podStartE2EDuration="1.839746578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:52:23.83703085 +0000 UTC m=+2470.006382125" watchObservedRunningTime="2026-07-21 12:52:23.839746578 +0000 UTC m=+2470.009097860" Jul 21 12:52:33.300449 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:33.300408 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:33.300869 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:33.300463 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:33.313034 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:33.313004 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:52:33.878401 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:52:33.878374 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:53:02.865115 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:02.865083 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj"] Jul 21 12:53:02.875131 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:02.875105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:03.762067 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:03.762031 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487"] Jul 21 12:53:03.781901 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:03.781876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:04.538589 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:04.538545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:04.639824 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:04.639779 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:14.373191 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:14.373095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:17.885161 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:17.885119 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" podUID="1e7c5a84-af42-4a0a-aa84-88046bf5af8b" containerName="main" containerID="cri-o://89f3d689f1202265e91dfc695f764fba15e310fc621f247327da20cc6b0b5957" gracePeriod=45 Jul 21 12:53:18.068118 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.068085 2559 generic.go:356] "Generic (PLEG): container finished" podID="1e7c5a84-af42-4a0a-aa84-88046bf5af8b" containerID="89f3d689f1202265e91dfc695f764fba15e310fc621f247327da20cc6b0b5957" exitCode=0 Jul 21 12:53:18.068281 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.068141 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" event={"ID":"1e7c5a84-af42-4a0a-aa84-88046bf5af8b","Type":"ContainerDied","Data":"89f3d689f1202265e91dfc695f764fba15e310fc621f247327da20cc6b0b5957"} Jul 21 12:53:18.131631 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.131608 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:53:18.175910 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.175832 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbhgw\" (UniqueName: \"kubernetes.io/projected/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-kube-api-access-zbhgw\") pod \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " Jul 21 12:53:18.175910 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.175885 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-dshm\") pod \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " Jul 21 12:53:18.176139 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.175997 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tls-certs\") pod \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " Jul 21 12:53:18.176139 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.176069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tmp-dir\") pod \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " Jul 21 12:53:18.176139 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.176095 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-model-cache\") pod \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " Jul 21 12:53:18.176309 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.176145 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-home\") pod \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\" (UID: \"1e7c5a84-af42-4a0a-aa84-88046bf5af8b\") " Jul 21 12:53:18.176423 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.176381 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e7c5a84-af42-4a0a-aa84-88046bf5af8b" (UID: "1e7c5a84-af42-4a0a-aa84-88046bf5af8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:18.176489 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.176402 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e7c5a84-af42-4a0a-aa84-88046bf5af8b" (UID: "1e7c5a84-af42-4a0a-aa84-88046bf5af8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:18.176489 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.176417 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-home" (OuterVolumeSpecName: "home") pod "1e7c5a84-af42-4a0a-aa84-88046bf5af8b" (UID: "1e7c5a84-af42-4a0a-aa84-88046bf5af8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:18.176606 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.176490 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:18.178141 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.178109 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-dshm" (OuterVolumeSpecName: "dshm") pod "1e7c5a84-af42-4a0a-aa84-88046bf5af8b" (UID: "1e7c5a84-af42-4a0a-aa84-88046bf5af8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:18.178253 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.178177 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e7c5a84-af42-4a0a-aa84-88046bf5af8b" (UID: "1e7c5a84-af42-4a0a-aa84-88046bf5af8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:53:18.178311 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.178247 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-kube-api-access-zbhgw" (OuterVolumeSpecName: "kube-api-access-zbhgw") pod "1e7c5a84-af42-4a0a-aa84-88046bf5af8b" (UID: "1e7c5a84-af42-4a0a-aa84-88046bf5af8b"). InnerVolumeSpecName "kube-api-access-zbhgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:53:18.277295 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.277263 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhgw\" (UniqueName: \"kubernetes.io/projected/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-kube-api-access-zbhgw\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:18.277295 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.277289 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:18.277295 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.277298 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:18.277511 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.277309 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:18.277511 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.277316 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7c5a84-af42-4a0a-aa84-88046bf5af8b-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:18.782296 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.782261 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" podUID="67ed5331-ce9e-47d4-a64d-f1daf8de4a89" containerName="main" containerID="cri-o://2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0" gracePeriod=45 Jul 21 12:53:18.989316 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:18.989281 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:19.029325 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.029304 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:53:19.074465 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.074383 2559 generic.go:356] "Generic (PLEG): container finished" podID="67ed5331-ce9e-47d4-a64d-f1daf8de4a89" containerID="2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0" exitCode=0 Jul 21 12:53:19.074465 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.074420 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" Jul 21 12:53:19.074690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.074471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" event={"ID":"67ed5331-ce9e-47d4-a64d-f1daf8de4a89","Type":"ContainerDied","Data":"2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0"} Jul 21 12:53:19.074690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.074515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487" event={"ID":"67ed5331-ce9e-47d4-a64d-f1daf8de4a89","Type":"ContainerDied","Data":"c090bdd3062ba04746822395821c4f63c09acb3272cdd687edf13a0108670774"} Jul 21 12:53:19.074690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.074537 2559 scope.go:117] "RemoveContainer" containerID="2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0" Jul 21 12:53:19.075788 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.075768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" event={"ID":"1e7c5a84-af42-4a0a-aa84-88046bf5af8b","Type":"ContainerDied","Data":"dca339c7eaf0f496ba68ba337179f8d812811faf7be620808cee8640a2305093"} Jul 21 12:53:19.075854 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.075778 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj" Jul 21 12:53:19.082758 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.082743 2559 scope.go:117] "RemoveContainer" containerID="2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0" Jul 21 12:53:19.083029 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:53:19.083013 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0\": container with ID starting with 2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0 not found: ID does not exist" containerID="2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0" Jul 21 12:53:19.083091 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.083039 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0"} err="failed to get container status \"2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0\": rpc error: code = NotFound desc = could not find container \"2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0\": container with ID starting with 2a3e43105720753c635bd02ed0f817f0c78fcd787825e7937365628b501a6ec0 not found: ID does not exist" Jul 21 12:53:19.083091 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.083056 2559 scope.go:117] "RemoveContainer" containerID="89f3d689f1202265e91dfc695f764fba15e310fc621f247327da20cc6b0b5957" Jul 21 12:53:19.083901 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.083888 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rshrq\" (UniqueName: \"kubernetes.io/projected/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-kube-api-access-rshrq\") pod \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " Jul 21 12:53:19.083939 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.083921 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tmp-dir\") pod \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " Jul 21 12:53:19.084031 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.083991 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-dshm\") pod \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " Jul 21 12:53:19.084087 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.084075 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-home\") pod \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " Jul 21 12:53:19.084134 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.084112 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tls-certs\") pod \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " Jul 21 12:53:19.084188 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.084151 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-model-cache\") pod \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\" (UID: \"67ed5331-ce9e-47d4-a64d-f1daf8de4a89\") " Jul 21 12:53:19.084434 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.084241 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67ed5331-ce9e-47d4-a64d-f1daf8de4a89" (UID: "67ed5331-ce9e-47d4-a64d-f1daf8de4a89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:19.084434 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.084375 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-home" (OuterVolumeSpecName: "home") pod "67ed5331-ce9e-47d4-a64d-f1daf8de4a89" (UID: "67ed5331-ce9e-47d4-a64d-f1daf8de4a89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:19.084434 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.084427 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:19.084623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.084439 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-model-cache" (OuterVolumeSpecName: "model-cache") pod "67ed5331-ce9e-47d4-a64d-f1daf8de4a89" (UID: "67ed5331-ce9e-47d4-a64d-f1daf8de4a89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:19.086222 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.086199 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-dshm" (OuterVolumeSpecName: "dshm") pod "67ed5331-ce9e-47d4-a64d-f1daf8de4a89" (UID: "67ed5331-ce9e-47d4-a64d-f1daf8de4a89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:53:19.086301 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.086200 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-kube-api-access-rshrq" (OuterVolumeSpecName: "kube-api-access-rshrq") pod "67ed5331-ce9e-47d4-a64d-f1daf8de4a89" (UID: "67ed5331-ce9e-47d4-a64d-f1daf8de4a89"). InnerVolumeSpecName "kube-api-access-rshrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:53:19.086569 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.086553 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67ed5331-ce9e-47d4-a64d-f1daf8de4a89" (UID: "67ed5331-ce9e-47d4-a64d-f1daf8de4a89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:53:19.093409 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.093391 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj"] Jul 21 12:53:19.099044 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.099026 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1cb55b/test-model-1cb55b-kserve-664cb8f98f-6mqvj"] Jul 21 12:53:19.185533 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.185506 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:19.185533 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.185528 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:19.185533 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.185537 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:19.185744 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.185546 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:19.185744 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.185556 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshrq\" (UniqueName: \"kubernetes.io/projected/67ed5331-ce9e-47d4-a64d-f1daf8de4a89-kube-api-access-rshrq\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:53:19.396922 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.396897 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487"] Jul 21 12:53:19.400636 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.400613 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ded38a/test-model-ded38a-kserve-68d45948b7-8k487"] Jul 21 12:53:19.410843 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:19.410810 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:20.360977 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:20.359941 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e7c5a84-af42-4a0a-aa84-88046bf5af8b" path="/var/lib/kubelet/pods/1e7c5a84-af42-4a0a-aa84-88046bf5af8b/volumes" Jul 21 12:53:20.360977 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:20.360578 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ed5331-ce9e-47d4-a64d-f1daf8de4a89" path="/var/lib/kubelet/pods/67ed5331-ce9e-47d4-a64d-f1daf8de4a89/volumes" Jul 21 12:53:25.540992 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:25.540920 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:25.625202 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:25.625171 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:25.816122 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:25.816044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:26.045394 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:26.045364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:26.684882 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:26.684855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:26.788610 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:26.788580 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:36.622004 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:36.621939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:41.242842 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:41.242801 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:41.612261 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:41.612163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:47.526581 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:47.526547 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:47.628007 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:47.627975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:47.805385 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:47.805287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:48.004164 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:48.004127 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:48.386423 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:48.386382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:48.690022 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:48.689928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:49.605886 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:49.605841 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:52.126739 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:52.126708 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:52.210244 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:52.210210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:52.295413 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:52.295382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:52.563919 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:52.563862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:52.852111 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:52.852010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:52.951776 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:52.951746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:56.617087 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:56.617054 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:57.073611 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:57.073582 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:57.365791 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:57.365712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:53:57.706032 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:53:57.705924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:00.238690 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:00.238646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:00.468266 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:00.468219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:00.572018 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:00.571913 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:00.742868 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:00.742842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:01.098584 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:01.098550 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:01.199645 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:01.199607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:04.361872 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:04.361840 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:04.597047 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:04.597017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:13.243516 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:13.243480 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:13.309072 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:13.309040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:21.898978 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:21.898933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:23.430071 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:23.430030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:23.526161 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:23.526122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:28.527210 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:28.527172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:32.690000 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:32.689969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:33.039623 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:33.039593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:38.751924 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:38.751829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:38.820164 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:38.820137 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:39.009681 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:39.009579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:39.242450 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:39.242419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:39.885569 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:39.885536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:40.000251 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:40.000197 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:55.049615 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:55.049584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:59.459318 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:59.459278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:54:59.849794 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:54:59.849758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:05.810082 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:05.810044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:05.902710 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:05.902651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:06.177770 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:06.177674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:06.445087 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:06.444987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:06.657541 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:06.657513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:06.997544 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:06.997515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:07.251774 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:07.251678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:09.606011 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:09.605943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:09.693458 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:09.693423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:09.768066 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:09.768036 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:09.909157 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:09.909060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:10.285128 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:10.285095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:10.381540 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:10.381514 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:14.048123 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:14.048083 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:14.366193 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:14.366072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:14.671551 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:14.671460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:14.974913 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:14.974880 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:17.634552 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:17.634513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:17.742193 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:17.742165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:17.874988 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:17.874945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:18.613104 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:18.613070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:18.721213 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:18.721183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:22.093799 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:22.093770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:24.507305 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:24.507275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:24.608248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:24.608215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:31.182236 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.182184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:31.289020 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.288984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:31.882294 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.882266 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv"] Jul 21 12:55:31.882630 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.882616 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7c5a84-af42-4a0a-aa84-88046bf5af8b" containerName="main" Jul 21 12:55:31.882679 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.882631 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7c5a84-af42-4a0a-aa84-88046bf5af8b" containerName="main" Jul 21 12:55:31.882679 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.882640 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ed5331-ce9e-47d4-a64d-f1daf8de4a89" containerName="main" Jul 21 12:55:31.882679 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.882646 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ed5331-ce9e-47d4-a64d-f1daf8de4a89" containerName="main" Jul 21 12:55:31.882768 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.882721 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ed5331-ce9e-47d4-a64d-f1daf8de4a89" containerName="main" Jul 21 12:55:31.882768 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.882732 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7c5a84-af42-4a0a-aa84-88046bf5af8b" containerName="main" Jul 21 12:55:31.885844 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.885827 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:31.889171 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.889139 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-782207\"/\"default-dockercfg-phbbg\"" Jul 21 12:55:31.889294 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.889192 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-782207\"/\"auth-test-model-782207-kserve-self-signed-certs\"" Jul 21 12:55:31.889294 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.889240 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-782207\"/\"kube-root-ca.crt\"" Jul 21 12:55:31.889386 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.889292 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-782207\"/\"openshift-service-ca.crt\"" Jul 21 12:55:31.896825 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.896798 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv"] Jul 21 12:55:31.909695 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:31.909670 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:32.023031 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.022998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-tmp-dir\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.023205 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.023043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-model-cache\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.023205 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.023090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-home\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.023205 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.023118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jltw\" (UniqueName: \"kubernetes.io/projected/30050170-85df-43fc-bf4d-c019c2af4f10-kube-api-access-4jltw\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.023205 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.023146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-dshm\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.023205 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.023162 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30050170-85df-43fc-bf4d-c019c2af4f10-tls-certs\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.123754 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.123720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-model-cache\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.123942 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.123766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-home\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.123942 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.123809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jltw\" (UniqueName: \"kubernetes.io/projected/30050170-85df-43fc-bf4d-c019c2af4f10-kube-api-access-4jltw\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.123942 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.123836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-dshm\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.123942 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.123857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30050170-85df-43fc-bf4d-c019c2af4f10-tls-certs\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.124206 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.123972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-tmp-dir\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.124265 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.124212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-model-cache\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.124265 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.124227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-home\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.124377 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.124331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-tmp-dir\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.126199 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.126170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-dshm\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.126347 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.126322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30050170-85df-43fc-bf4d-c019c2af4f10-tls-certs\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.133414 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.133351 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jltw\" (UniqueName: \"kubernetes.io/projected/30050170-85df-43fc-bf4d-c019c2af4f10-kube-api-access-4jltw\") pod \"auth-test-model-782207-kserve-5cd97864dc-mthmv\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.194611 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.194564 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:32.325168 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.325141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv"] Jul 21 12:55:32.327250 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:55:32.327225 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30050170_85df_43fc_bf4d_c019c2af4f10.slice/crio-f2b316c27b604362286f6b11e5ab6ccb96f495d4e19578cd0c59f3dce73e6dd8 WatchSource:0}: Error finding container f2b316c27b604362286f6b11e5ab6ccb96f495d4e19578cd0c59f3dce73e6dd8: Status 404 returned error can't find the container with id f2b316c27b604362286f6b11e5ab6ccb96f495d4e19578cd0c59f3dce73e6dd8 Jul 21 12:55:32.662818 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.662777 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" event={"ID":"30050170-85df-43fc-bf4d-c019c2af4f10","Type":"ContainerStarted","Data":"d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8"} Jul 21 12:55:32.662818 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.662824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" event={"ID":"30050170-85df-43fc-bf4d-c019c2af4f10","Type":"ContainerStarted","Data":"f2b316c27b604362286f6b11e5ab6ccb96f495d4e19578cd0c59f3dce73e6dd8"} Jul 21 12:55:32.682064 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:32.682012 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" podStartSLOduration=1.681998488 podStartE2EDuration="1.681998488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:55:32.680212686 +0000 UTC m=+2658.849563962" watchObservedRunningTime="2026-07-21 12:55:32.681998488 +0000 UTC m=+2658.851349765" Jul 21 12:55:40.975227 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:40.975185 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:41.073675 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:41.073643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:55:42.194702 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:42.194667 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:42.195112 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:42.194726 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:42.207507 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:42.207482 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:55:42.717186 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:55:42.717160 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:56:13.035650 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.035565 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx"] Jul 21 12:56:13.038790 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.038770 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.042572 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.042435 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"auth-test-model-48842b-kserve-self-signed-certs\"" Jul 21 12:56:13.042572 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.042450 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"openshift-service-ca.crt\"" Jul 21 12:56:13.042572 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.042495 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"kube-root-ca.crt\"" Jul 21 12:56:13.042782 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.042542 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"default-dockercfg-9pq49\"" Jul 21 12:56:13.055053 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.055028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:56:13.062793 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.062770 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx"] Jul 21 12:56:13.088515 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.088493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tls-certs\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.088634 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.088540 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-home\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.088634 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.088575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-dshm\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.088634 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.088598 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-model-cache\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.088634 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.088623 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tmp-dir\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.088843 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.088749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgm99\" (UniqueName: \"kubernetes.io/projected/203e6d3d-cd05-4b96-a557-1ff370bd20a9-kube-api-access-vgm99\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.189809 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.189769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgm99\" (UniqueName: \"kubernetes.io/projected/203e6d3d-cd05-4b96-a557-1ff370bd20a9-kube-api-access-vgm99\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190026 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.189924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tls-certs\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190026 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.190001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-home\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.190030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-dshm\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.190052 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-model-cache\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.190102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tmp-dir\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190546 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.190522 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tmp-dir\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.190544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-home\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.190628 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.190569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-model-cache\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.192247 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.192227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-dshm\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.192333 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.192324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tls-certs\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.211008 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.210987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgm99\" (UniqueName: \"kubernetes.io/projected/203e6d3d-cd05-4b96-a557-1ff370bd20a9-kube-api-access-vgm99\") pod \"auth-test-model-48842b-kserve-88c66767f-4nllx\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.349076 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.348998 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:13.496384 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.496358 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx"] Jul 21 12:56:13.498862 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:56:13.498832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod203e6d3d_cd05_4b96_a557_1ff370bd20a9.slice/crio-d08f911efb64f58bb71e5f2b11f5c9f4b9f67cdfda9f9f98f29dbad372b65582 WatchSource:0}: Error finding container d08f911efb64f58bb71e5f2b11f5c9f4b9f67cdfda9f9f98f29dbad372b65582: Status 404 returned error can't find the container with id d08f911efb64f58bb71e5f2b11f5c9f4b9f67cdfda9f9f98f29dbad372b65582 Jul 21 12:56:13.842358 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.842316 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" event={"ID":"203e6d3d-cd05-4b96-a557-1ff370bd20a9","Type":"ContainerStarted","Data":"b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57"} Jul 21 12:56:13.842521 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:13.842366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" event={"ID":"203e6d3d-cd05-4b96-a557-1ff370bd20a9","Type":"ContainerStarted","Data":"d08f911efb64f58bb71e5f2b11f5c9f4b9f67cdfda9f9f98f29dbad372b65582"} Jul 21 12:56:14.440503 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:14.440473 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:56:14.450018 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:14.449994 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 12:56:23.349806 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:23.349764 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:23.350330 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:23.349914 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:23.362436 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:23.362412 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:56:23.380090 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:23.380037 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" podStartSLOduration=10.380021239 podStartE2EDuration="10.380021239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:56:13.869566595 +0000 UTC m=+2700.038917871" watchObservedRunningTime="2026-07-21 12:56:23.380021239 +0000 UTC m=+2709.549372515" Jul 21 12:56:23.896452 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:56:23.896424 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:57:39.491261 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:57:39.491183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:57:43.274922 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:57:43.274885 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:57:43.374737 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:57:43.374702 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:57:49.567781 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:57:49.567747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:57:49.642440 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:57:49.642409 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:58:18.894861 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:18.894826 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv"] Jul 21 12:58:18.917345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:18.917317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:58:19.796321 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:19.796291 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx"] Jul 21 12:58:19.825808 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:19.825782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:58:28.381281 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.381246 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9"] Jul 21 12:58:28.384613 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.384593 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.386825 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.386804 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-782207\"/\"sub-test-model-782207-kserve-self-signed-certs\"" Jul 21 12:58:28.394356 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.394326 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9"] Jul 21 12:58:28.467997 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.467970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-dshm\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.468134 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.468005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-model-cache\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.468134 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.468023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmggr\" (UniqueName: \"kubernetes.io/projected/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-kube-api-access-qmggr\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.468134 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.468049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-home\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.468134 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.468118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tmp-dir\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.468376 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.468146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tls-certs\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.487192 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.487170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:58:28.568757 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.568730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-dshm\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.568875 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.568763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-model-cache\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.568875 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.568782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmggr\" (UniqueName: \"kubernetes.io/projected/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-kube-api-access-qmggr\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.568875 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.568800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-home\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.569080 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.568923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tmp-dir\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.569080 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.568994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tls-certs\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.569238 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.569220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-home\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.569343 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.569324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-model-cache\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.569417 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.569395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tmp-dir\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.570864 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.570844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-dshm\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.571308 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.571289 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tls-certs\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.576896 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.576872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmggr\" (UniqueName: \"kubernetes.io/projected/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-kube-api-access-qmggr\") pod \"sub-test-model-782207-kserve-687ddbf579-92nj9\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.693808 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.693741 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:28.825859 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:28.825806 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9"] Jul 21 12:58:28.832397 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:58:28.832366 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5e6996c_a3ad_4feb_ab7b_17fccbdedb6c.slice/crio-b7d52229c9ec7593b2f4144cb4be768e97cec7bb8592add6fbea7e6a9e5ccfdd WatchSource:0}: Error finding container b7d52229c9ec7593b2f4144cb4be768e97cec7bb8592add6fbea7e6a9e5ccfdd: Status 404 returned error can't find the container with id b7d52229c9ec7593b2f4144cb4be768e97cec7bb8592add6fbea7e6a9e5ccfdd Jul 21 12:58:29.430171 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:29.430135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" event={"ID":"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c","Type":"ContainerStarted","Data":"43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64"} Jul 21 12:58:29.430171 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:29.430179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" event={"ID":"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c","Type":"ContainerStarted","Data":"b7d52229c9ec7593b2f4144cb4be768e97cec7bb8592add6fbea7e6a9e5ccfdd"} Jul 21 12:58:29.449094 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:29.449047 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" podStartSLOduration=1.4490281409999999 podStartE2EDuration="1.449028141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:58:29.447247084 +0000 UTC m=+2835.616598361" watchObservedRunningTime="2026-07-21 12:58:29.449028141 +0000 UTC m=+2835.618379434" Jul 21 12:58:33.914570 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:33.914533 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" podUID="30050170-85df-43fc-bf4d-c019c2af4f10" containerName="main" containerID="cri-o://d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8" gracePeriod=45 Jul 21 12:58:34.178295 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.178241 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:58:34.221186 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221162 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-tmp-dir\") pod \"30050170-85df-43fc-bf4d-c019c2af4f10\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " Jul 21 12:58:34.221310 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221220 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30050170-85df-43fc-bf4d-c019c2af4f10-tls-certs\") pod \"30050170-85df-43fc-bf4d-c019c2af4f10\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " Jul 21 12:58:34.221310 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221252 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-dshm\") pod \"30050170-85df-43fc-bf4d-c019c2af4f10\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " Jul 21 12:58:34.221310 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221286 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jltw\" (UniqueName: \"kubernetes.io/projected/30050170-85df-43fc-bf4d-c019c2af4f10-kube-api-access-4jltw\") pod \"30050170-85df-43fc-bf4d-c019c2af4f10\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " Jul 21 12:58:34.221434 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221350 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-model-cache\") pod \"30050170-85df-43fc-bf4d-c019c2af4f10\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " Jul 21 12:58:34.221434 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221384 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-home\") pod \"30050170-85df-43fc-bf4d-c019c2af4f10\" (UID: \"30050170-85df-43fc-bf4d-c019c2af4f10\") " Jul 21 12:58:34.221541 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221436 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30050170-85df-43fc-bf4d-c019c2af4f10" (UID: "30050170-85df-43fc-bf4d-c019c2af4f10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:34.221668 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-model-cache" (OuterVolumeSpecName: "model-cache") pod "30050170-85df-43fc-bf4d-c019c2af4f10" (UID: "30050170-85df-43fc-bf4d-c019c2af4f10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:34.221754 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221733 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-home" (OuterVolumeSpecName: "home") pod "30050170-85df-43fc-bf4d-c019c2af4f10" (UID: "30050170-85df-43fc-bf4d-c019c2af4f10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:34.221818 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221775 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:34.221818 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.221794 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:34.223543 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.223517 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30050170-85df-43fc-bf4d-c019c2af4f10-kube-api-access-4jltw" (OuterVolumeSpecName: "kube-api-access-4jltw") pod "30050170-85df-43fc-bf4d-c019c2af4f10" (UID: "30050170-85df-43fc-bf4d-c019c2af4f10"). InnerVolumeSpecName "kube-api-access-4jltw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:58:34.223640 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.223572 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-dshm" (OuterVolumeSpecName: "dshm") pod "30050170-85df-43fc-bf4d-c019c2af4f10" (UID: "30050170-85df-43fc-bf4d-c019c2af4f10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:34.223713 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.223698 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30050170-85df-43fc-bf4d-c019c2af4f10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30050170-85df-43fc-bf4d-c019c2af4f10" (UID: "30050170-85df-43fc-bf4d-c019c2af4f10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:58:34.322816 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.322792 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:34.322816 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.322814 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30050170-85df-43fc-bf4d-c019c2af4f10-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:34.322816 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.322823 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30050170-85df-43fc-bf4d-c019c2af4f10-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:34.323064 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.322833 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jltw\" (UniqueName: \"kubernetes.io/projected/30050170-85df-43fc-bf4d-c019c2af4f10-kube-api-access-4jltw\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:34.454865 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.454775 2559 generic.go:356] "Generic (PLEG): container finished" podID="30050170-85df-43fc-bf4d-c019c2af4f10" containerID="d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8" exitCode=0 Jul 21 12:58:34.454865 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.454817 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" Jul 21 12:58:34.454865 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.454822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" event={"ID":"30050170-85df-43fc-bf4d-c019c2af4f10","Type":"ContainerDied","Data":"d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8"} Jul 21 12:58:34.454865 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.454847 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv" event={"ID":"30050170-85df-43fc-bf4d-c019c2af4f10","Type":"ContainerDied","Data":"f2b316c27b604362286f6b11e5ab6ccb96f495d4e19578cd0c59f3dce73e6dd8"} Jul 21 12:58:34.454865 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.454863 2559 scope.go:117] "RemoveContainer" containerID="d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8" Jul 21 12:58:34.464020 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.464000 2559 scope.go:117] "RemoveContainer" containerID="d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8" Jul 21 12:58:34.464294 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:58:34.464275 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8\": container with ID starting with d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8 not found: ID does not exist" containerID="d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8" Jul 21 12:58:34.464380 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.464300 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8"} err="failed to get container status \"d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8\": rpc error: code = NotFound desc = could not find container \"d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8\": container with ID starting with d6b4ef12ffcc019dd33efa64130db6ad50b6548febc7851666a9b4b5c75e3ed8 not found: ID does not exist" Jul 21 12:58:34.472920 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.472899 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv"] Jul 21 12:58:34.477167 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.477147 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-782207/auth-test-model-782207-kserve-5cd97864dc-mthmv"] Jul 21 12:58:34.815655 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:34.815613 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" podUID="203e6d3d-cd05-4b96-a557-1ff370bd20a9" containerName="main" containerID="cri-o://b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57" gracePeriod=45 Jul 21 12:58:35.064120 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.064099 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:58:35.130755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.130689 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgm99\" (UniqueName: \"kubernetes.io/projected/203e6d3d-cd05-4b96-a557-1ff370bd20a9-kube-api-access-vgm99\") pod \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " Jul 21 12:58:35.130755 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.130747 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-dshm\") pod \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " Jul 21 12:58:35.130913 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.130772 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tmp-dir\") pod \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " Jul 21 12:58:35.130991 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.130931 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-model-cache\") pod \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " Jul 21 12:58:35.130991 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.130983 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "203e6d3d-cd05-4b96-a557-1ff370bd20a9" (UID: "203e6d3d-cd05-4b96-a557-1ff370bd20a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:35.131133 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.131050 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-home\") pod \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " Jul 21 12:58:35.131133 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.131110 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tls-certs\") pod \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\" (UID: \"203e6d3d-cd05-4b96-a557-1ff370bd20a9\") " Jul 21 12:58:35.131243 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.131189 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "203e6d3d-cd05-4b96-a557-1ff370bd20a9" (UID: "203e6d3d-cd05-4b96-a557-1ff370bd20a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:35.131315 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.131295 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-home" (OuterVolumeSpecName: "home") pod "203e6d3d-cd05-4b96-a557-1ff370bd20a9" (UID: "203e6d3d-cd05-4b96-a557-1ff370bd20a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:35.131487 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.131462 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:35.131541 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.131491 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:35.131541 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.131502 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:35.133197 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.133172 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203e6d3d-cd05-4b96-a557-1ff370bd20a9-kube-api-access-vgm99" (OuterVolumeSpecName: "kube-api-access-vgm99") pod "203e6d3d-cd05-4b96-a557-1ff370bd20a9" (UID: "203e6d3d-cd05-4b96-a557-1ff370bd20a9"). InnerVolumeSpecName "kube-api-access-vgm99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:58:35.133302 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.133191 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "203e6d3d-cd05-4b96-a557-1ff370bd20a9" (UID: "203e6d3d-cd05-4b96-a557-1ff370bd20a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:58:35.133302 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.133233 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-dshm" (OuterVolumeSpecName: "dshm") pod "203e6d3d-cd05-4b96-a557-1ff370bd20a9" (UID: "203e6d3d-cd05-4b96-a557-1ff370bd20a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:58:35.232099 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.232068 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/203e6d3d-cd05-4b96-a557-1ff370bd20a9-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:35.232099 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.232095 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203e6d3d-cd05-4b96-a557-1ff370bd20a9-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:35.232099 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.232106 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgm99\" (UniqueName: \"kubernetes.io/projected/203e6d3d-cd05-4b96-a557-1ff370bd20a9-kube-api-access-vgm99\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:58:35.466773 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.466678 2559 generic.go:356] "Generic (PLEG): container finished" podID="203e6d3d-cd05-4b96-a557-1ff370bd20a9" containerID="b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57" exitCode=0 Jul 21 12:58:35.466773 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.466736 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" Jul 21 12:58:35.466773 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.466753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" event={"ID":"203e6d3d-cd05-4b96-a557-1ff370bd20a9","Type":"ContainerDied","Data":"b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57"} Jul 21 12:58:35.467067 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.466797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx" event={"ID":"203e6d3d-cd05-4b96-a557-1ff370bd20a9","Type":"ContainerDied","Data":"d08f911efb64f58bb71e5f2b11f5c9f4b9f67cdfda9f9f98f29dbad372b65582"} Jul 21 12:58:35.467067 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.466819 2559 scope.go:117] "RemoveContainer" containerID="b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57" Jul 21 12:58:35.475529 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.475514 2559 scope.go:117] "RemoveContainer" containerID="b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57" Jul 21 12:58:35.475771 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:58:35.475750 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57\": container with ID starting with b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57 not found: ID does not exist" containerID="b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57" Jul 21 12:58:35.475810 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.475780 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57"} err="failed to get container status \"b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57\": rpc error: code = NotFound desc = could not find container \"b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57\": container with ID starting with b8695f683cce72def8b66a32cef4b5e7f72ff71fe1969b8830ccaaefbc80af57 not found: ID does not exist" Jul 21 12:58:35.487173 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.487152 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx"] Jul 21 12:58:35.492627 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:35.492604 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/auth-test-model-48842b-kserve-88c66767f-4nllx"] Jul 21 12:58:36.358069 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:36.358035 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203e6d3d-cd05-4b96-a557-1ff370bd20a9" path="/var/lib/kubelet/pods/203e6d3d-cd05-4b96-a557-1ff370bd20a9/volumes" Jul 21 12:58:36.358459 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:36.358443 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30050170-85df-43fc-bf4d-c019c2af4f10" path="/var/lib/kubelet/pods/30050170-85df-43fc-bf4d-c019c2af4f10/volumes" Jul 21 12:58:38.694772 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:38.694742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:38.694772 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:38.694786 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:38.706998 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:38.706977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:39.496336 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:39.496309 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:58:46.490446 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.490410 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s"] Jul 21 12:58:46.490909 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.490891 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30050170-85df-43fc-bf4d-c019c2af4f10" containerName="main" Jul 21 12:58:46.491014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.490914 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="30050170-85df-43fc-bf4d-c019c2af4f10" containerName="main" Jul 21 12:58:46.491014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.490971 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203e6d3d-cd05-4b96-a557-1ff370bd20a9" containerName="main" Jul 21 12:58:46.491014 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.490982 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="203e6d3d-cd05-4b96-a557-1ff370bd20a9" containerName="main" Jul 21 12:58:46.491181 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.491084 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="203e6d3d-cd05-4b96-a557-1ff370bd20a9" containerName="main" Jul 21 12:58:46.491181 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.491098 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="30050170-85df-43fc-bf4d-c019c2af4f10" containerName="main" Jul 21 12:58:46.494229 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.494206 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.497462 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.497431 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"openshift-service-ca.crt\"" Jul 21 12:58:46.497587 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.497497 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"sub-test-model-48842b-kserve-self-signed-certs\"" Jul 21 12:58:46.497587 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.497428 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"default-dockercfg-9pq49\"" Jul 21 12:58:46.497587 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.497549 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"kube-root-ca.crt\"" Jul 21 12:58:46.504564 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.504541 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s"] Jul 21 12:58:46.528704 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.528676 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:58:46.627426 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.627395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-tmp-dir\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.627562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.627446 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-home\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.627562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.627465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-dshm\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.627562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.627527 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-model-cache\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.627562 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.627560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq98n\" (UniqueName: \"kubernetes.io/projected/c68336d6-dd19-4c7c-a28b-249a452f76a3-kube-api-access-lq98n\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.627737 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.627602 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68336d6-dd19-4c7c-a28b-249a452f76a3-tls-certs\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.728575 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-tmp-dir\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.728730 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-home\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.728730 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-dshm\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.728730 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-model-cache\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.728898 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq98n\" (UniqueName: \"kubernetes.io/projected/c68336d6-dd19-4c7c-a28b-249a452f76a3-kube-api-access-lq98n\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.728898 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68336d6-dd19-4c7c-a28b-249a452f76a3-tls-certs\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.728898 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-tmp-dir\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.729092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.728984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-home\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.729092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.729042 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-model-cache\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.730980 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.730942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-dshm\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.731180 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.731165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68336d6-dd19-4c7c-a28b-249a452f76a3-tls-certs\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.736846 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.736826 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq98n\" (UniqueName: \"kubernetes.io/projected/c68336d6-dd19-4c7c-a28b-249a452f76a3-kube-api-access-lq98n\") pod \"sub-test-model-48842b-kserve-f4d579d47-pgb2s\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.803814 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.803758 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:46.862584 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.862558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:58:46.936767 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.936642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s"] Jul 21 12:58:46.941192 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:46.941168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:58:46.941796 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:58:46.941770 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc68336d6_dd19_4c7c_a28b_249a452f76a3.slice/crio-5629794bc6b22157d92be3eb091193cebaa508eaf868688c30411372e4c2f302 WatchSource:0}: Error finding container 5629794bc6b22157d92be3eb091193cebaa508eaf868688c30411372e4c2f302: Status 404 returned error can't find the container with id 5629794bc6b22157d92be3eb091193cebaa508eaf868688c30411372e4c2f302 Jul 21 12:58:47.528777 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:47.528736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" event={"ID":"c68336d6-dd19-4c7c-a28b-249a452f76a3","Type":"ContainerStarted","Data":"5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df"} Jul 21 12:58:47.529279 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:47.528786 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" event={"ID":"c68336d6-dd19-4c7c-a28b-249a452f76a3","Type":"ContainerStarted","Data":"5629794bc6b22157d92be3eb091193cebaa508eaf868688c30411372e4c2f302"} Jul 21 12:58:47.547989 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:47.547913 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" podStartSLOduration=1.5478972020000001 podStartE2EDuration="1.547897202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:58:47.546175696 +0000 UTC m=+2853.715526995" watchObservedRunningTime="2026-07-21 12:58:47.547897202 +0000 UTC m=+2853.717248478" Jul 21 12:58:56.804176 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:56.804143 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:56.804767 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:56.804189 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:56.816659 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:56.816636 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:58:57.584734 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:58:57.584709 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:59:33.266221 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:33.266184 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9"] Jul 21 12:59:33.307497 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:33.307469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:59:34.065897 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:34.065867 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s"] Jul 21 12:59:35.050679 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:35.050625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:59:47.899969 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:47.899917 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28"] Jul 21 12:59:47.903213 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:47.903193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:47.905607 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:47.905588 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-782207\"/\"rate-test-model-782207-kserve-self-signed-certs\"" Jul 21 12:59:47.912378 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:47.912358 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28"] Jul 21 12:59:47.979055 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:47.979020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 12:59:48.045182 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.045156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-dshm\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.045345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.045214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-home\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.045345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.045238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8306e8-0161-4111-b97d-6a5f15f2967e-tls-certs\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.045345 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.045305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqlrw\" (UniqueName: \"kubernetes.io/projected/dc8306e8-0161-4111-b97d-6a5f15f2967e-kube-api-access-mqlrw\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.045478 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.045355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-tmp-dir\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.045515 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.045492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-model-cache\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.146490 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-model-cache\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.146631 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-dshm\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.146631 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-home\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.146631 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8306e8-0161-4111-b97d-6a5f15f2967e-tls-certs\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.146631 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqlrw\" (UniqueName: \"kubernetes.io/projected/dc8306e8-0161-4111-b97d-6a5f15f2967e-kube-api-access-mqlrw\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.146631 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-tmp-dir\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.146927 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-model-cache\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.147075 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.146980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-home\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.147075 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.147022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-tmp-dir\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.148855 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.148823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-dshm\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.149070 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.149054 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8306e8-0161-4111-b97d-6a5f15f2967e-tls-certs\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.155655 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.155603 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqlrw\" (UniqueName: \"kubernetes.io/projected/dc8306e8-0161-4111-b97d-6a5f15f2967e-kube-api-access-mqlrw\") pod \"rate-test-model-782207-kserve-75bbd976c-dlb28\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.212965 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.212928 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:48.285629 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.285560 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" podUID="d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" containerName="main" containerID="cri-o://43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64" gracePeriod=45 Jul 21 12:59:48.359551 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.359522 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28"] Jul 21 12:59:48.361843 ip-10-0-135-97 kubenswrapper[2559]: W0721 12:59:48.361821 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc8306e8_0161_4111_b97d_6a5f15f2967e.slice/crio-36110437220d45d6ec893163fb9a24622e15d25bd9fa2b1fdee53732660e4fde WatchSource:0}: Error finding container 36110437220d45d6ec893163fb9a24622e15d25bd9fa2b1fdee53732660e4fde: Status 404 returned error can't find the container with id 36110437220d45d6ec893163fb9a24622e15d25bd9fa2b1fdee53732660e4fde Jul 21 12:59:48.524276 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.524248 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:59:48.652646 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652615 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tmp-dir\") pod \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " Jul 21 12:59:48.652821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652665 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-home\") pod \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " Jul 21 12:59:48.652821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652722 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-model-cache\") pod \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " Jul 21 12:59:48.652821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652747 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tls-certs\") pod \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " Jul 21 12:59:48.652821 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652802 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-dshm\") pod \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " Jul 21 12:59:48.653097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652850 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmggr\" (UniqueName: \"kubernetes.io/projected/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-kube-api-access-qmggr\") pod \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\" (UID: \"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c\") " Jul 21 12:59:48.653097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652915 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" (UID: "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:48.653097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.652932 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-home" (OuterVolumeSpecName: "home") pod "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" (UID: "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:48.653097 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.653011 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" (UID: "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:48.653287 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.653217 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:48.653287 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.653235 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:48.653287 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.653249 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:48.654857 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.654836 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-kube-api-access-qmggr" (OuterVolumeSpecName: "kube-api-access-qmggr") pod "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" (UID: "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c"). InnerVolumeSpecName "kube-api-access-qmggr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:59:48.655272 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.655254 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-dshm" (OuterVolumeSpecName: "dshm") pod "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" (UID: "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:48.655319 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.655300 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" (UID: "d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:59:48.753969 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.753922 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:48.753969 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.753944 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:48.753969 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.753974 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmggr\" (UniqueName: \"kubernetes.io/projected/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c-kube-api-access-qmggr\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:48.795857 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.795821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" event={"ID":"dc8306e8-0161-4111-b97d-6a5f15f2967e","Type":"ContainerStarted","Data":"3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981"} Jul 21 12:59:48.796039 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.795862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" event={"ID":"dc8306e8-0161-4111-b97d-6a5f15f2967e","Type":"ContainerStarted","Data":"36110437220d45d6ec893163fb9a24622e15d25bd9fa2b1fdee53732660e4fde"} Jul 21 12:59:48.797032 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.797010 2559 generic.go:356] "Generic (PLEG): container finished" podID="d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" containerID="43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64" exitCode=0 Jul 21 12:59:48.797148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.797053 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" event={"ID":"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c","Type":"ContainerDied","Data":"43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64"} Jul 21 12:59:48.797148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.797058 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" Jul 21 12:59:48.797148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.797074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9" event={"ID":"d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c","Type":"ContainerDied","Data":"b7d52229c9ec7593b2f4144cb4be768e97cec7bb8592add6fbea7e6a9e5ccfdd"} Jul 21 12:59:48.797148 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.797089 2559 scope.go:117] "RemoveContainer" containerID="43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64" Jul 21 12:59:48.805112 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.805095 2559 scope.go:117] "RemoveContainer" containerID="43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64" Jul 21 12:59:48.805353 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:59:48.805335 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64\": container with ID starting with 43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64 not found: ID does not exist" containerID="43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64" Jul 21 12:59:48.805399 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.805361 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64"} err="failed to get container status \"43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64\": rpc error: code = NotFound desc = could not find container \"43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64\": container with ID starting with 43f635a3795b6fe18c43c7cd284975a6872956c550d1b2b47e087f2458f41d64 not found: ID does not exist" Jul 21 12:59:48.812882 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.812844 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" podStartSLOduration=1.812832875 podStartE2EDuration="1.812832875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 12:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 12:59:48.812237285 +0000 UTC m=+2914.981588560" watchObservedRunningTime="2026-07-21 12:59:48.812832875 +0000 UTC m=+2914.982184182" Jul 21 12:59:48.824279 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.824249 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9"] Jul 21 12:59:48.827601 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:48.827581 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-782207/sub-test-model-782207-kserve-687ddbf579-92nj9"] Jul 21 12:59:49.086935 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.086845 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" podUID="c68336d6-dd19-4c7c-a28b-249a452f76a3" containerName="main" containerID="cri-o://5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df" gracePeriod=45 Jul 21 12:59:49.328667 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.328641 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:59:49.459009 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.458926 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-tmp-dir\") pod \"c68336d6-dd19-4c7c-a28b-249a452f76a3\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " Jul 21 12:59:49.459009 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.458980 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq98n\" (UniqueName: \"kubernetes.io/projected/c68336d6-dd19-4c7c-a28b-249a452f76a3-kube-api-access-lq98n\") pod \"c68336d6-dd19-4c7c-a28b-249a452f76a3\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " Jul 21 12:59:49.459009 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459006 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-home\") pod \"c68336d6-dd19-4c7c-a28b-249a452f76a3\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " Jul 21 12:59:49.459266 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459029 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-model-cache\") pod \"c68336d6-dd19-4c7c-a28b-249a452f76a3\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " Jul 21 12:59:49.459266 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459079 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68336d6-dd19-4c7c-a28b-249a452f76a3-tls-certs\") pod \"c68336d6-dd19-4c7c-a28b-249a452f76a3\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " Jul 21 12:59:49.459266 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459124 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-dshm\") pod \"c68336d6-dd19-4c7c-a28b-249a452f76a3\" (UID: \"c68336d6-dd19-4c7c-a28b-249a452f76a3\") " Jul 21 12:59:49.459266 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459218 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c68336d6-dd19-4c7c-a28b-249a452f76a3" (UID: "c68336d6-dd19-4c7c-a28b-249a452f76a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:49.459468 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459288 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-home" (OuterVolumeSpecName: "home") pod "c68336d6-dd19-4c7c-a28b-249a452f76a3" (UID: "c68336d6-dd19-4c7c-a28b-249a452f76a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:49.459468 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459304 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "c68336d6-dd19-4c7c-a28b-249a452f76a3" (UID: "c68336d6-dd19-4c7c-a28b-249a452f76a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:49.459576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459523 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:49.459576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459542 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:49.459576 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.459555 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:49.461248 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.461227 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68336d6-dd19-4c7c-a28b-249a452f76a3-kube-api-access-lq98n" (OuterVolumeSpecName: "kube-api-access-lq98n") pod "c68336d6-dd19-4c7c-a28b-249a452f76a3" (UID: "c68336d6-dd19-4c7c-a28b-249a452f76a3"). InnerVolumeSpecName "kube-api-access-lq98n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 12:59:49.461560 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.461538 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-dshm" (OuterVolumeSpecName: "dshm") pod "c68336d6-dd19-4c7c-a28b-249a452f76a3" (UID: "c68336d6-dd19-4c7c-a28b-249a452f76a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 12:59:49.461699 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.461551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68336d6-dd19-4c7c-a28b-249a452f76a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c68336d6-dd19-4c7c-a28b-249a452f76a3" (UID: "c68336d6-dd19-4c7c-a28b-249a452f76a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 12:59:49.559931 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.559905 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68336d6-dd19-4c7c-a28b-249a452f76a3-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:49.559931 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.559927 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68336d6-dd19-4c7c-a28b-249a452f76a3-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:49.560092 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.559936 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq98n\" (UniqueName: \"kubernetes.io/projected/c68336d6-dd19-4c7c-a28b-249a452f76a3-kube-api-access-lq98n\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 12:59:49.806276 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.806241 2559 generic.go:356] "Generic (PLEG): container finished" podID="c68336d6-dd19-4c7c-a28b-249a452f76a3" containerID="5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df" exitCode=0 Jul 21 12:59:49.806436 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.806284 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" Jul 21 12:59:49.806436 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.806320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" event={"ID":"c68336d6-dd19-4c7c-a28b-249a452f76a3","Type":"ContainerDied","Data":"5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df"} Jul 21 12:59:49.806436 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.806370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s" event={"ID":"c68336d6-dd19-4c7c-a28b-249a452f76a3","Type":"ContainerDied","Data":"5629794bc6b22157d92be3eb091193cebaa508eaf868688c30411372e4c2f302"} Jul 21 12:59:49.806436 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.806387 2559 scope.go:117] "RemoveContainer" containerID="5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df" Jul 21 12:59:49.814969 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.814927 2559 scope.go:117] "RemoveContainer" containerID="5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df" Jul 21 12:59:49.815201 ip-10-0-135-97 kubenswrapper[2559]: E0721 12:59:49.815183 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df\": container with ID starting with 5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df not found: ID does not exist" containerID="5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df" Jul 21 12:59:49.815247 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.815210 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df"} err="failed to get container status \"5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df\": rpc error: code = NotFound desc = could not find container \"5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df\": container with ID starting with 5ef03851a710f306f291031879b74a1c925143560ee3dbb22fdf9cf9a66166df not found: ID does not exist" Jul 21 12:59:49.828973 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.828626 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s"] Jul 21 12:59:49.830389 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:49.830371 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/sub-test-model-48842b-kserve-f4d579d47-pgb2s"] Jul 21 12:59:50.357757 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:50.357727 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c68336d6-dd19-4c7c-a28b-249a452f76a3" path="/var/lib/kubelet/pods/c68336d6-dd19-4c7c-a28b-249a452f76a3/volumes" Jul 21 12:59:50.358144 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:50.358126 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" path="/var/lib/kubelet/pods/d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c/volumes" Jul 21 12:59:58.213728 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:58.213690 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:58.214229 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:58.213749 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:58.226447 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:58.226422 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 12:59:58.860388 ip-10-0-135-97 kubenswrapper[2559]: I0721 12:59:58.860360 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 13:00:10.988575 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.988543 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst"] Jul 21 13:00:10.989107 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.988885 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" containerName="main" Jul 21 13:00:10.989107 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.988898 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" containerName="main" Jul 21 13:00:10.989107 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.988917 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68336d6-dd19-4c7c-a28b-249a452f76a3" containerName="main" Jul 21 13:00:10.989107 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.988923 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68336d6-dd19-4c7c-a28b-249a452f76a3" containerName="main" Jul 21 13:00:10.989342 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.989165 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e6996c-a3ad-4feb-ab7b-17fccbdedb6c" containerName="main" Jul 21 13:00:10.989342 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.989183 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c68336d6-dd19-4c7c-a28b-249a452f76a3" containerName="main" Jul 21 13:00:10.992475 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.992455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:10.994968 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.994928 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"openshift-service-ca.crt\"" Jul 21 13:00:10.995107 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.995051 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"default-dockercfg-9pq49\"" Jul 21 13:00:10.995889 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.995871 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"kube-root-ca.crt\"" Jul 21 13:00:10.996037 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:10.996019 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-48842b\"/\"rate-test-model-48842b-kserve-self-signed-certs\"" Jul 21 13:00:11.003142 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.003120 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst"] Jul 21 13:00:11.046702 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.046663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qnf\" (UniqueName: \"kubernetes.io/projected/f7895204-94aa-4f6b-84bc-fedab171fcd6-kube-api-access-w7qnf\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.046873 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.046713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-dshm\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.046873 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.046798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-tmp-dir\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.046873 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.046842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-model-cache\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.047025 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.046872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7895204-94aa-4f6b-84bc-fedab171fcd6-tls-certs\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.047025 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.047008 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-home\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.147744 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.147707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-home\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.147975 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.147759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qnf\" (UniqueName: \"kubernetes.io/projected/f7895204-94aa-4f6b-84bc-fedab171fcd6-kube-api-access-w7qnf\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.147975 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.147781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-dshm\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.147975 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.147899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-tmp-dir\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.147975 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.147943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-model-cache\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.148192 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.148000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7895204-94aa-4f6b-84bc-fedab171fcd6-tls-certs\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.148235 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.148189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-home\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.148279 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.148263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-tmp-dir\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.148412 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.148384 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-model-cache\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.150126 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.150105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-dshm\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.150412 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.150394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7895204-94aa-4f6b-84bc-fedab171fcd6-tls-certs\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.156756 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.156734 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qnf\" (UniqueName: \"kubernetes.io/projected/f7895204-94aa-4f6b-84bc-fedab171fcd6-kube-api-access-w7qnf\") pod \"rate-test-model-48842b-kserve-6484f64c59-brkst\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.303444 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.303363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:11.447724 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.447688 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst"] Jul 21 13:00:11.450705 ip-10-0-135-97 kubenswrapper[2559]: W0721 13:00:11.450671 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7895204_94aa_4f6b_84bc_fedab171fcd6.slice/crio-45b58dc0843f5cdca07c4d7dfa65378cb2c74d32a1732484814dd3d417320087 WatchSource:0}: Error finding container 45b58dc0843f5cdca07c4d7dfa65378cb2c74d32a1732484814dd3d417320087: Status 404 returned error can't find the container with id 45b58dc0843f5cdca07c4d7dfa65378cb2c74d32a1732484814dd3d417320087 Jul 21 13:00:11.486881 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.486852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:00:11.905931 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.905897 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" event={"ID":"f7895204-94aa-4f6b-84bc-fedab171fcd6","Type":"ContainerStarted","Data":"e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e"} Jul 21 13:00:11.905931 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.905934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" event={"ID":"f7895204-94aa-4f6b-84bc-fedab171fcd6","Type":"ContainerStarted","Data":"45b58dc0843f5cdca07c4d7dfa65378cb2c74d32a1732484814dd3d417320087"} Jul 21 13:00:11.925063 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:11.925014 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" podStartSLOduration=1.9250001490000002 podStartE2EDuration="1.925000149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 13:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:00:11.922744232 +0000 UTC m=+2938.092095521" watchObservedRunningTime="2026-07-21 13:00:11.925000149 +0000 UTC m=+2938.094351425" Jul 21 13:00:21.303608 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:21.303575 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:21.303608 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:21.303616 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:21.315968 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:21.315931 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:21.965098 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:21.965072 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:00:25.299527 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:25.299500 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:00:25.415686 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:25.415652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:00:32.053436 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:32.053401 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:00:33.404453 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:33.404423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:00:41.771098 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:41.771013 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:00:41.871919 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:41.871886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:00:49.741728 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:00:49.741680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:02.790979 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:02.790935 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28"] Jul 21 13:01:02.813229 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:02.813199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:03.861927 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:03.861895 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst"] Jul 21 13:01:03.883108 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:03.883079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:14.466392 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:14.466363 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 13:01:14.475693 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:14.475675 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qs5s_eef530bc-aba7-4b53-93d6-c8818dcffe5a/ovn-acl-logging/0.log" Jul 21 13:01:15.443259 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:15.443221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:17.810691 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:17.810656 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" podUID="dc8306e8-0161-4111-b97d-6a5f15f2967e" containerName="main" containerID="cri-o://3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981" gracePeriod=45 Jul 21 13:01:18.063378 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.063321 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 13:01:18.207080 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.207049 2559 generic.go:356] "Generic (PLEG): container finished" podID="dc8306e8-0161-4111-b97d-6a5f15f2967e" containerID="3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981" exitCode=0 Jul 21 13:01:18.207255 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.207087 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" event={"ID":"dc8306e8-0161-4111-b97d-6a5f15f2967e","Type":"ContainerDied","Data":"3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981"} Jul 21 13:01:18.207255 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.207090 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" Jul 21 13:01:18.207255 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.207113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28" event={"ID":"dc8306e8-0161-4111-b97d-6a5f15f2967e","Type":"ContainerDied","Data":"36110437220d45d6ec893163fb9a24622e15d25bd9fa2b1fdee53732660e4fde"} Jul 21 13:01:18.207255 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.207130 2559 scope.go:117] "RemoveContainer" containerID="3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981" Jul 21 13:01:18.215349 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.215332 2559 scope.go:117] "RemoveContainer" containerID="3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981" Jul 21 13:01:18.215608 ip-10-0-135-97 kubenswrapper[2559]: E0721 13:01:18.215589 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981\": container with ID starting with 3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981 not found: ID does not exist" containerID="3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981" Jul 21 13:01:18.215656 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.215618 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981"} err="failed to get container status \"3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981\": rpc error: code = NotFound desc = could not find container \"3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981\": container with ID starting with 3816bb3378879d1195762a3e7aa66363b1b4689886ad9c958aa321cfc747a981 not found: ID does not exist" Jul 21 13:01:18.222588 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222570 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8306e8-0161-4111-b97d-6a5f15f2967e-tls-certs\") pod \"dc8306e8-0161-4111-b97d-6a5f15f2967e\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " Jul 21 13:01:18.222687 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222595 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-tmp-dir\") pod \"dc8306e8-0161-4111-b97d-6a5f15f2967e\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " Jul 21 13:01:18.222687 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222611 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-model-cache\") pod \"dc8306e8-0161-4111-b97d-6a5f15f2967e\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " Jul 21 13:01:18.222687 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222658 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqlrw\" (UniqueName: \"kubernetes.io/projected/dc8306e8-0161-4111-b97d-6a5f15f2967e-kube-api-access-mqlrw\") pod \"dc8306e8-0161-4111-b97d-6a5f15f2967e\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " Jul 21 13:01:18.222819 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222748 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-home\") pod \"dc8306e8-0161-4111-b97d-6a5f15f2967e\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " Jul 21 13:01:18.222819 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222774 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-dshm\") pod \"dc8306e8-0161-4111-b97d-6a5f15f2967e\" (UID: \"dc8306e8-0161-4111-b97d-6a5f15f2967e\") " Jul 21 13:01:18.222922 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222847 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc8306e8-0161-4111-b97d-6a5f15f2967e" (UID: "dc8306e8-0161-4111-b97d-6a5f15f2967e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:18.222922 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.222899 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc8306e8-0161-4111-b97d-6a5f15f2967e" (UID: "dc8306e8-0161-4111-b97d-6a5f15f2967e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:18.223052 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.223029 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-home" (OuterVolumeSpecName: "home") pod "dc8306e8-0161-4111-b97d-6a5f15f2967e" (UID: "dc8306e8-0161-4111-b97d-6a5f15f2967e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:18.223114 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.223081 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.223114 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.223093 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.224768 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.224743 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8306e8-0161-4111-b97d-6a5f15f2967e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc8306e8-0161-4111-b97d-6a5f15f2967e" (UID: "dc8306e8-0161-4111-b97d-6a5f15f2967e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:01:18.224861 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.224748 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-dshm" (OuterVolumeSpecName: "dshm") pod "dc8306e8-0161-4111-b97d-6a5f15f2967e" (UID: "dc8306e8-0161-4111-b97d-6a5f15f2967e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:18.224861 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.224817 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8306e8-0161-4111-b97d-6a5f15f2967e-kube-api-access-mqlrw" (OuterVolumeSpecName: "kube-api-access-mqlrw") pod "dc8306e8-0161-4111-b97d-6a5f15f2967e" (UID: "dc8306e8-0161-4111-b97d-6a5f15f2967e"). InnerVolumeSpecName "kube-api-access-mqlrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:01:18.324026 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.323970 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqlrw\" (UniqueName: \"kubernetes.io/projected/dc8306e8-0161-4111-b97d-6a5f15f2967e-kube-api-access-mqlrw\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.324026 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.323991 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.324026 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.324001 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc8306e8-0161-4111-b97d-6a5f15f2967e-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.324026 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.324010 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8306e8-0161-4111-b97d-6a5f15f2967e-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:18.522251 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.522220 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28"] Jul 21 13:01:18.528120 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.528096 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-782207/rate-test-model-782207-kserve-75bbd976c-dlb28"] Jul 21 13:01:18.881553 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:18.881499 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" podUID="f7895204-94aa-4f6b-84bc-fedab171fcd6" containerName="main" containerID="cri-o://e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e" gracePeriod=45 Jul 21 13:01:19.133169 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.133105 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:01:19.213215 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.213180 2559 generic.go:356] "Generic (PLEG): container finished" podID="f7895204-94aa-4f6b-84bc-fedab171fcd6" containerID="e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e" exitCode=0 Jul 21 13:01:19.213379 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.213223 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" Jul 21 13:01:19.213379 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.213259 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" event={"ID":"f7895204-94aa-4f6b-84bc-fedab171fcd6","Type":"ContainerDied","Data":"e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e"} Jul 21 13:01:19.213379 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.213305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst" event={"ID":"f7895204-94aa-4f6b-84bc-fedab171fcd6","Type":"ContainerDied","Data":"45b58dc0843f5cdca07c4d7dfa65378cb2c74d32a1732484814dd3d417320087"} Jul 21 13:01:19.213379 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.213328 2559 scope.go:117] "RemoveContainer" containerID="e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e" Jul 21 13:01:19.221178 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.221163 2559 scope.go:117] "RemoveContainer" containerID="e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e" Jul 21 13:01:19.221424 ip-10-0-135-97 kubenswrapper[2559]: E0721 13:01:19.221406 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e\": container with ID starting with e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e not found: ID does not exist" containerID="e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e" Jul 21 13:01:19.221495 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.221430 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e"} err="failed to get container status \"e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e\": rpc error: code = NotFound desc = could not find container \"e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e\": container with ID starting with e4178f128ca72882682f9799e87beb5f90135405314827e42ddb09c29f836b1e not found: ID does not exist" Jul 21 13:01:19.231734 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.231715 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-dshm\") pod \"f7895204-94aa-4f6b-84bc-fedab171fcd6\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " Jul 21 13:01:19.231795 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.231743 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7895204-94aa-4f6b-84bc-fedab171fcd6-tls-certs\") pod \"f7895204-94aa-4f6b-84bc-fedab171fcd6\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " Jul 21 13:01:19.231795 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.231776 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-model-cache\") pod \"f7895204-94aa-4f6b-84bc-fedab171fcd6\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " Jul 21 13:01:19.231900 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.231799 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7qnf\" (UniqueName: \"kubernetes.io/projected/f7895204-94aa-4f6b-84bc-fedab171fcd6-kube-api-access-w7qnf\") pod \"f7895204-94aa-4f6b-84bc-fedab171fcd6\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " Jul 21 13:01:19.231900 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.231843 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-tmp-dir\") pod \"f7895204-94aa-4f6b-84bc-fedab171fcd6\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " Jul 21 13:01:19.231900 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.231876 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-home\") pod \"f7895204-94aa-4f6b-84bc-fedab171fcd6\" (UID: \"f7895204-94aa-4f6b-84bc-fedab171fcd6\") " Jul 21 13:01:19.232079 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.232013 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7895204-94aa-4f6b-84bc-fedab171fcd6" (UID: "f7895204-94aa-4f6b-84bc-fedab171fcd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:19.232157 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.232141 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-model-cache\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:19.232214 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.232131 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7895204-94aa-4f6b-84bc-fedab171fcd6" (UID: "f7895204-94aa-4f6b-84bc-fedab171fcd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:19.232214 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.232159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-home" (OuterVolumeSpecName: "home") pod "f7895204-94aa-4f6b-84bc-fedab171fcd6" (UID: "f7895204-94aa-4f6b-84bc-fedab171fcd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:19.233773 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.233752 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-dshm" (OuterVolumeSpecName: "dshm") pod "f7895204-94aa-4f6b-84bc-fedab171fcd6" (UID: "f7895204-94aa-4f6b-84bc-fedab171fcd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 13:01:19.233934 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.233907 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7895204-94aa-4f6b-84bc-fedab171fcd6-kube-api-access-w7qnf" (OuterVolumeSpecName: "kube-api-access-w7qnf") pod "f7895204-94aa-4f6b-84bc-fedab171fcd6" (UID: "f7895204-94aa-4f6b-84bc-fedab171fcd6"). InnerVolumeSpecName "kube-api-access-w7qnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 13:01:19.234032 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.234012 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7895204-94aa-4f6b-84bc-fedab171fcd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7895204-94aa-4f6b-84bc-fedab171fcd6" (UID: "f7895204-94aa-4f6b-84bc-fedab171fcd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 13:01:19.332698 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.332673 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-home\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:19.332698 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.332692 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-dshm\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:19.332698 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.332702 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7895204-94aa-4f6b-84bc-fedab171fcd6-tls-certs\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:19.332929 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.332711 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qnf\" (UniqueName: \"kubernetes.io/projected/f7895204-94aa-4f6b-84bc-fedab171fcd6-kube-api-access-w7qnf\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:19.332929 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.332719 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7895204-94aa-4f6b-84bc-fedab171fcd6-tmp-dir\") on node \"ip-10-0-135-97.ec2.internal\" DevicePath \"\"" Jul 21 13:01:19.538887 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.538861 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst"] Jul 21 13:01:19.544706 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:19.544681 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-48842b/rate-test-model-48842b-kserve-6484f64c59-brkst"] Jul 21 13:01:20.358388 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:20.358352 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc8306e8-0161-4111-b97d-6a5f15f2967e" path="/var/lib/kubelet/pods/dc8306e8-0161-4111-b97d-6a5f15f2967e/volumes" Jul 21 13:01:20.358767 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:20.358736 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7895204-94aa-4f6b-84bc-fedab171fcd6" path="/var/lib/kubelet/pods/f7895204-94aa-4f6b-84bc-fedab171fcd6/volumes" Jul 21 13:01:23.501781 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:23.501746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:23.602394 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:23.602363 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:33.600385 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:33.600346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:38.066095 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:38.066058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:38.479826 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:38.479794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:44.309587 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:44.309557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:44.409719 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:44.409692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:44.583026 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:44.582927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:44.807150 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:44.807115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:45.444403 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:45.444371 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:45.544487 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:45.544457 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:01:55.563683 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:01:55.563651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:00.013628 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:00.013597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:00.372850 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:00.372776 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:06.286973 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:06.286873 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:06.498591 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:06.498560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:06.600387 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:06.600291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:06.757423 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:06.757389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:13.086855 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:13.086829 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f85c6b8d-5qdxn_4506e84a-8330-468d-a823-549b20f8cd0d/copy-manifests/0.log" Jul 21 13:02:13.100073 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:13.100047 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f85c6b8d-5qdxn_4506e84a-8330-468d-a823-549b20f8cd0d/manager/0.log" Jul 21 13:02:15.828763 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:15.828737 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8mtr9_2f0cceaf-5da2-493c-8d4c-0099fa790c90/kuadrant-console-plugin/0.log" Jul 21 13:02:16.058829 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:16.058797 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-khz5t_226f6c60-cdce-48a7-8b29-ea86356b7b54/manager/0.log" Jul 21 13:02:16.927005 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:16.926976 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc49b7cdf-cf48x_ae191bd0-0927-4f31-8a15-f3d9b7be4934/kube-auth-proxy/0.log" Jul 21 13:02:17.301080 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:17.301046 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-p299v_0bb69899-55f7-4121-a3e1-d6cef741bdc1/payload-pre-processing/0.log" Jul 21 13:02:18.033530 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:18.033504 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-n95kv_1d6d1621-fdf2-4e70-8138-d335eeedb12b/main/0.log" Jul 21 13:02:18.145753 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:18.145726 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-krgfn_f7814600-3a73-45d7-b8f2-dc48b3e81fe0/main/0.log" Jul 21 13:02:18.470847 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:18.470775 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-lbsvr_82e35fbb-4aa3-4f8e-b791-b4b1668fe043/main/0.log" Jul 21 13:02:24.177194 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:24.177158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:24.313777 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:24.313745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" limit=1024 Jul 21 13:02:27.670227 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:27.670199 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xmpjs_426a4044-78b3-421b-abd3-85f56961434a/global-pull-secret-syncer/0.log" Jul 21 13:02:27.751002 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:27.750975 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9jkd6_eae41061-6724-4683-b038-5495c849fa21/konnectivity-agent/0.log" Jul 21 13:02:27.913213 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:27.913188 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-97.ec2.internal_e306b331cd25c79c60bbf3257cbc2b52/haproxy/0.log" Jul 21 13:02:33.228307 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:33.228283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8mtr9_2f0cceaf-5da2-493c-8d4c-0099fa790c90/kuadrant-console-plugin/0.log" Jul 21 13:02:33.439473 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:33.439439 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-khz5t_226f6c60-cdce-48a7-8b29-ea86356b7b54/manager/0.log" Jul 21 13:02:35.564070 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:35.564042 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lnx9q_b14c2e65-60cb-4354-9cb9-ff7c88a96db3/kube-state-metrics/0.log" Jul 21 13:02:35.588788 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:35.588755 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lnx9q_b14c2e65-60cb-4354-9cb9-ff7c88a96db3/kube-rbac-proxy-main/0.log" Jul 21 13:02:35.611710 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:35.611688 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lnx9q_b14c2e65-60cb-4354-9cb9-ff7c88a96db3/kube-rbac-proxy-self/0.log" Jul 21 13:02:35.669188 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:35.669169 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-854t9_317fced9-b067-4e34-b9ba-02b6fb876f09/monitoring-plugin/0.log" Jul 21 13:02:35.860737 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:35.860699 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2w2d_677b3c04-ae20-4b8b-9a30-1d06ce7e99cd/node-exporter/0.log" Jul 21 13:02:35.889644 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:35.889618 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2w2d_677b3c04-ae20-4b8b-9a30-1d06ce7e99cd/kube-rbac-proxy/0.log" Jul 21 13:02:35.913466 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:35.913445 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2w2d_677b3c04-ae20-4b8b-9a30-1d06ce7e99cd/init-textfile/0.log" Jul 21 13:02:36.035335 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.035298 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1cbddd0-55dd-489d-acae-fbc5148404a3/prometheus/0.log" Jul 21 13:02:36.058216 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.058196 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1cbddd0-55dd-489d-acae-fbc5148404a3/config-reloader/0.log" Jul 21 13:02:36.084723 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.084699 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1cbddd0-55dd-489d-acae-fbc5148404a3/thanos-sidecar/0.log" Jul 21 13:02:36.120694 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.120642 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1cbddd0-55dd-489d-acae-fbc5148404a3/kube-rbac-proxy-web/0.log" Jul 21 13:02:36.144538 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.144519 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1cbddd0-55dd-489d-acae-fbc5148404a3/kube-rbac-proxy/0.log" Jul 21 13:02:36.169309 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.169292 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1cbddd0-55dd-489d-acae-fbc5148404a3/kube-rbac-proxy-thanos/0.log" Jul 21 13:02:36.192322 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.192307 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1cbddd0-55dd-489d-acae-fbc5148404a3/init-config-reloader/0.log" Jul 21 13:02:36.222670 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.222652 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8gqtj_b3d72b58-3205-47f5-87d2-96427d9c667f/prometheus-operator/0.log" Jul 21 13:02:36.244333 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.244317 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8gqtj_b3d72b58-3205-47f5-87d2-96427d9c667f/kube-rbac-proxy/0.log" Jul 21 13:02:36.283350 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:36.283320 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-658pv_564e8037-d7c8-4a4d-af58-a02ba80792be/prometheus-operator-admission-webhook/0.log" Jul 21 13:02:37.773875 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.773839 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm"] Jul 21 13:02:37.774420 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.774343 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc8306e8-0161-4111-b97d-6a5f15f2967e" containerName="main" Jul 21 13:02:37.774420 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.774364 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc8306e8-0161-4111-b97d-6a5f15f2967e" containerName="main" Jul 21 13:02:37.774420 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.774384 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7895204-94aa-4f6b-84bc-fedab171fcd6" containerName="main" Jul 21 13:02:37.774420 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.774392 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7895204-94aa-4f6b-84bc-fedab171fcd6" containerName="main" Jul 21 13:02:37.774632 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.774472 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc8306e8-0161-4111-b97d-6a5f15f2967e" containerName="main" Jul 21 13:02:37.774632 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.774486 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7895204-94aa-4f6b-84bc-fedab171fcd6" containerName="main" Jul 21 13:02:37.777839 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.777813 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.780277 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.780254 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nqg59\"/\"kube-root-ca.crt\"" Jul 21 13:02:37.781099 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.781078 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nqg59\"/\"default-dockercfg-6wm6t\"" Jul 21 13:02:37.781244 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.781084 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nqg59\"/\"openshift-service-ca.crt\"" Jul 21 13:02:37.792615 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.792596 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm"] Jul 21 13:02:37.879808 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.879780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-sys\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.879959 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.879830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-proc\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.879959 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.879859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h75bn\" (UniqueName: \"kubernetes.io/projected/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-kube-api-access-h75bn\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.880043 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.880002 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-podres\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.880077 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.880054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-lib-modules\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.980938 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.980906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-sys\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981102 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.980984 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-proc\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981102 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.981006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75bn\" (UniqueName: \"kubernetes.io/projected/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-kube-api-access-h75bn\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981102 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.981027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-sys\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981102 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.981072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-podres\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981242 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.981103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-proc\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981242 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.981110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-lib-modules\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981242 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.981208 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-lib-modules\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.981242 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.981211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-podres\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:37.989195 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:37.989175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h75bn\" (UniqueName: \"kubernetes.io/projected/ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42-kube-api-access-h75bn\") pod \"perf-node-gather-daemonset-d9dkm\" (UID: \"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42\") " pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:38.087512 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:38.087451 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:38.211971 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:38.211924 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm"] Jul 21 13:02:38.214127 ip-10-0-135-97 kubenswrapper[2559]: W0721 13:02:38.214095 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podddd68b3e_6497_4ee4_b5a0_d98ca24b8a42.slice/crio-12f4f7ec033485df78dae1b57e529f33d381d9b72419dc4f59dc6957b766f443 WatchSource:0}: Error finding container 12f4f7ec033485df78dae1b57e529f33d381d9b72419dc4f59dc6957b766f443: Status 404 returned error can't find the container with id 12f4f7ec033485df78dae1b57e529f33d381d9b72419dc4f59dc6957b766f443 Jul 21 13:02:38.558989 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:38.558933 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" event={"ID":"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42","Type":"ContainerStarted","Data":"204539c6680b1c7bcb5dc0a96cc22b011ab8b4a2352fb46267e4697c3fe421ca"} Jul 21 13:02:38.559151 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:38.559011 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" event={"ID":"ddd68b3e-6497-4ee4-b5a0-d98ca24b8a42","Type":"ContainerStarted","Data":"12f4f7ec033485df78dae1b57e529f33d381d9b72419dc4f59dc6957b766f443"} Jul 21 13:02:38.559151 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:38.559091 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:38.577251 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:38.577209 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" podStartSLOduration=1.577197417 podStartE2EDuration="1.577197417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 13:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 13:02:38.575516701 +0000 UTC m=+3084.744867977" watchObservedRunningTime="2026-07-21 13:02:38.577197417 +0000 UTC m=+3084.746548694" Jul 21 13:02:40.683028 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:40.682992 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nbhbz_a413545d-fdc9-45c1-b1b9-79cb9965cfa7/dns/0.log" Jul 21 13:02:40.704635 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:40.704607 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nbhbz_a413545d-fdc9-45c1-b1b9-79cb9965cfa7/kube-rbac-proxy/0.log" Jul 21 13:02:40.860041 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:40.860016 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j5tm5_206118a1-01d3-44ca-894f-0c229a671f20/dns-node-resolver/0.log" Jul 21 13:02:41.467054 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:41.467021 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-644f4d8c7f-t5c6h_b21f7106-a24b-4cfd-b6d0-9f505af3368e/registry/0.log" Jul 21 13:02:41.510003 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:41.509979 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmb77_2b1f89f7-ac5a-4452-8a2f-1e74a36c5e8d/node-ca/0.log" Jul 21 13:02:42.782482 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:42.782454 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc49b7cdf-cf48x_ae191bd0-0927-4f31-8a15-f3d9b7be4934/kube-auth-proxy/0.log" Jul 21 13:02:42.912301 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:42.912274 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-p299v_0bb69899-55f7-4121-a3e1-d6cef741bdc1/payload-pre-processing/0.log" Jul 21 13:02:43.752141 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:43.752113 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4jt2_680cbf07-e123-4188-8052-8d691df7d859/serve-healthcheck-canary/0.log" Jul 21 13:02:44.342542 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:44.342518 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kxw7m_e9cdf497-3071-44fc-8a5e-9dbf895ba5db/kube-rbac-proxy/0.log" Jul 21 13:02:44.364114 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:44.364096 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kxw7m_e9cdf497-3071-44fc-8a5e-9dbf895ba5db/exporter/0.log" Jul 21 13:02:44.386189 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:44.386170 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kxw7m_e9cdf497-3071-44fc-8a5e-9dbf895ba5db/extractor/0.log" Jul 21 13:02:44.573170 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:44.573149 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nqg59/perf-node-gather-daemonset-d9dkm" Jul 21 13:02:47.819811 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:47.819782 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f85c6b8d-5qdxn_4506e84a-8330-468d-a823-549b20f8cd0d/manager/0.log" Jul 21 13:02:47.843055 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:47.843029 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f85c6b8d-5qdxn_4506e84a-8330-468d-a823-549b20f8cd0d/copy-manifests/0.log" Jul 21 13:02:49.016349 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:49.016321 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc49b7cdf-cf48x_ae191bd0-0927-4f31-8a15-f3d9b7be4934/kube-auth-proxy/0.log" Jul 21 13:02:49.127698 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:49.127669 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-p299v_0bb69899-55f7-4121-a3e1-d6cef741bdc1/payload-pre-processing/0.log" Jul 21 13:02:49.285369 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:49.285340 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c65cd9d66-wzrkf_044ea10b-97f0-4968-8a6b-ddd21255288f/manager/0.log" Jul 21 13:02:50.660799 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:50.660773 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fc49b7cdf-cf48x_ae191bd0-0927-4f31-8a15-f3d9b7be4934/kube-auth-proxy/0.log" Jul 21 13:02:50.761454 ip-10-0-135-97 kubenswrapper[2559]: I0721 13:02:50.761430 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-p299v_0bb69899-55f7-4121-a3e1-d6cef741bdc1/payload-pre-processing/0.log"