Jul 21 14:30:25.807474 ip-10-0-130-112 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 14:30:25.807508 ip-10-0-130-112 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 14:30:25.807519 ip-10-0-130-112 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 14:30:25.807874 ip-10-0-130-112 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 14:30:36.056249 ip-10-0-130-112 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 14:30:36.056272 ip-10-0-130-112 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e047ac08a85541029fa65c0fa21a1347 -- Jul 21 14:32:35.387490 ip-10-0-130-112 systemd[1]: Starting Kubernetes Kubelet... Jul 21 14:32:36.626964 ip-10-0-130-112 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:32:36.626964 ip-10-0-130-112 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 14:32:36.626964 ip-10-0-130-112 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:32:36.626964 ip-10-0-130-112 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 14:32:36.626964 ip-10-0-130-112 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:32:36.629964 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.629869 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 14:32:36.634894 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634875 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:32:36.634894 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634893 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634897 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634902 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634907 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634910 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634913 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634915 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634918 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634921 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634924 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634929 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634933 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634937 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634941 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634944 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634947 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634950 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634953 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634957 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634960 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:32:36.634967 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634963 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634967 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634971 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634975 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634978 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634982 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634985 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634989 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634991 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634994 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.634997 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635000 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635003 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635007 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635011 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635014 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635017 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635020 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635022 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:32:36.635485 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635025 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635027 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635030 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635033 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635035 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635038 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635041 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635044 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635047 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635050 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635053 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635056 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635058 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635061 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635064 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635067 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635070 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635074 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635077 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:32:36.635959 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635079 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635082 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635085 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635088 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635091 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635094 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635096 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635099 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635102 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635104 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635107 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635110 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635113 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635116 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635119 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635121 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635124 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635127 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635129 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635133 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:32:36.636536 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635136 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635139 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635142 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635144 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635147 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635150 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635152 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635155 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635159 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635162 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635165 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635168 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635171 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635174 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635177 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635179 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635182 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635185 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635187 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635190 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:32:36.637960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635986 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635993 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.635996 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636000 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636003 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636006 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636009 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636012 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636015 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636020 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636025 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636028 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636031 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636034 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636039 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636043 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636046 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636050 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636053 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:32:36.638880 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636056 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636060 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636063 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636066 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636069 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636072 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636075 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636078 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636081 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636084 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636087 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636090 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636092 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636095 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636098 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636101 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636103 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636106 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636109 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636112 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:32:36.639775 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636114 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636117 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636120 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636123 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636125 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636128 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636131 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636133 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636136 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636139 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636141 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636144 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636147 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636150 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636153 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636156 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636158 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636161 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636163 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636166 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:32:36.640671 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636169 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636172 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636174 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636177 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636181 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636183 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636186 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636189 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636191 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636194 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636197 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636200 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636202 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636205 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636207 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636210 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636214 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636217 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636220 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:32:36.641179 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636223 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636225 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636228 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636231 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636233 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636236 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636241 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636244 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636248 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636252 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636255 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636258 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636261 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636263 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636266 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636269 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636271 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636274 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636277 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636280 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:32:36.641773 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.636283 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638356 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638374 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638381 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638386 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638391 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638396 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638401 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638406 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638411 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638416 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638421 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638425 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638432 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638437 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638441 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638446 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638450 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638455 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638459 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:32:36.642509 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638464 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638469 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638474 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638482 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638488 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638493 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638499 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638505 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638510 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638515 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638519 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638524 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638528 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638534 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638538 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638544 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638551 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638556 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638561 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638565 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:32:36.643148 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638570 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638574 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638579 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638583 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638588 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638592 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638599 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638606 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638611 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638616 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638620 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638626 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638634 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638641 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638646 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638651 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638656 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638661 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638665 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:32:36.643690 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638670 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638674 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638679 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638684 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638688 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638693 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638701 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638706 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638711 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638715 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638720 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638724 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638729 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638733 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638738 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638743 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638747 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638752 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638756 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638760 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:32:36.644308 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638765 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638773 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638777 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638782 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638786 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638792 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638796 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638801 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638805 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638809 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638820 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638825 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638830 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638834 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638839 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638843 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638848 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638853 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638859 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638863 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:32:36.644859 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.638868 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.638984 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.638997 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639008 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639016 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639023 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639028 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639037 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639044 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639050 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639055 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639061 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639067 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639072 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639077 2559 flags.go:64] FLAG: --cgroup-root="" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639083 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639088 2559 flags.go:64] FLAG: --client-ca-file="" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639093 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639099 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639107 2559 flags.go:64] FLAG: --cluster-domain="" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639112 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639117 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639125 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639131 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 14:32:36.645404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639138 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639143 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639148 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639154 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639159 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639164 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639170 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639175 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639180 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639187 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639193 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639198 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639202 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639208 2559 flags.go:64] FLAG: --enable-server="true" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639213 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639221 2559 flags.go:64] FLAG: --event-burst="100" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639226 2559 flags.go:64] FLAG: --event-qps="50" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639231 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639236 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639240 2559 flags.go:64] FLAG: --eviction-hard="" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639247 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639252 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639257 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639263 2559 flags.go:64] FLAG: --eviction-soft="" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639268 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 14:32:36.646011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639273 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639278 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639283 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639308 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639314 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639321 2559 flags.go:64] FLAG: --feature-gates="" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639328 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639334 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639339 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639344 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639349 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639355 2559 flags.go:64] FLAG: --help="false" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639361 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-130-112.ec2.internal" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639366 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639372 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639377 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639383 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639389 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639394 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639399 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639405 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639410 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639415 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639421 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 14:32:36.646677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639426 2559 flags.go:64] FLAG: --kube-reserved="" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639431 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639436 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639441 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639446 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639451 2559 flags.go:64] FLAG: --lock-file="" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639456 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639461 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639467 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639476 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639481 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639487 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639493 2559 flags.go:64] FLAG: --logging-format="text" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639501 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639507 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639512 2559 flags.go:64] FLAG: --manifest-url="" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639517 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639524 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639530 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639544 2559 flags.go:64] FLAG: --max-pods="110" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639550 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639555 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639560 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639566 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639571 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 14:32:36.647439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639576 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639581 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639595 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639601 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639606 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639611 2559 flags.go:64] FLAG: --pod-cidr="" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639616 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639626 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639631 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639636 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639641 2559 flags.go:64] FLAG: --port="10250" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639646 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639651 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f9a4af86e08f7414" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639657 2559 flags.go:64] FLAG: --qos-reserved="" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639662 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639667 2559 flags.go:64] FLAG: --register-node="true" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639672 2559 flags.go:64] FLAG: --register-with-taints="" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639678 2559 flags.go:64] FLAG: --registry-burst="10" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639683 2559 flags.go:64] FLAG: --registry-qps="5" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639689 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639696 2559 flags.go:64] FLAG: --reserved-memory="" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639703 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639708 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639714 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639719 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 14:32:36.648086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639723 2559 flags.go:64] FLAG: --runonce="false" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639728 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639733 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639739 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639744 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639749 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639754 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639759 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639765 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639770 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639775 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639780 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639784 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639789 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639794 2559 flags.go:64] FLAG: --system-cgroups="" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639799 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639810 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639815 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639820 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639828 2559 flags.go:64] FLAG: --tls-min-version="" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639833 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639838 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639842 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639847 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639852 2559 flags.go:64] FLAG: --v="2" Jul 21 14:32:36.648797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639860 2559 flags.go:64] FLAG: --version="false" Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639867 2559 flags.go:64] FLAG: --vmodule="" Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639876 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.639882 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640083 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640090 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640096 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640102 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640107 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640112 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640117 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640124 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640129 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640134 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640138 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640143 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640148 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640156 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640162 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640168 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:32:36.649539 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640173 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640178 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640182 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640186 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640191 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640195 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640200 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640205 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640209 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640213 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640218 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640223 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640227 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640232 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640239 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640245 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640252 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640258 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640263 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640267 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:32:36.650042 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640272 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640277 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640283 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640306 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640312 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640316 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640321 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640326 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640330 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640335 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640339 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640343 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640348 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640353 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640358 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640362 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640367 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640372 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640376 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640381 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:32:36.650566 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640385 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640390 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640395 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640400 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640404 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640409 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640415 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640420 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640428 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640434 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640440 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640445 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640450 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640455 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640460 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640466 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640471 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640476 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640481 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:32:36.651093 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640486 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640491 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640495 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640500 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640504 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640509 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640513 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640518 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640523 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640528 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640532 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640537 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640541 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640546 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640550 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640554 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640559 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640564 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640568 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640574 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:32:36.651600 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640579 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640584 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640589 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.640594 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.641247 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.648900 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.648920 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.649853 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.649865 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649927 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649933 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649937 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649940 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649944 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649947 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:32:36.652115 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649951 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649954 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649958 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649961 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649964 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649968 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649970 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649974 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649977 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649980 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649983 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649985 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649988 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649991 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649995 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.649999 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650003 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650006 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650010 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:32:36.652570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650012 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650015 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650018 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650021 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650026 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650029 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650034 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650038 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650042 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650044 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650048 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650051 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650054 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650056 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650059 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650062 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650065 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650069 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650072 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:32:36.653053 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650075 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650078 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650081 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650084 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650087 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650090 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650092 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650095 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650098 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650101 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650104 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650106 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650109 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650112 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650115 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650118 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650121 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650124 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650136 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650139 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:32:36.653564 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650142 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650145 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650147 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650150 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650153 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650156 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650159 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650162 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650164 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650167 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650170 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650173 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650175 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650178 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650181 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650184 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650187 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650190 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650192 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:32:36.654069 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650195 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650199 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650203 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650206 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650209 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650213 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650216 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650219 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650222 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650225 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650229 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650233 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650237 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650240 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650243 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:32:36.654570 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650261 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.650267 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650410 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650416 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650419 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650422 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650425 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650428 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650432 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650435 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650438 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650441 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650445 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650447 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650450 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:32:36.654989 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650453 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650456 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650459 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650461 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650464 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650467 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650470 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650473 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650475 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650478 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650480 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650483 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650486 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650489 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650492 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650496 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650499 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650501 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650504 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650507 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:32:36.655418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650510 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650512 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650515 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650518 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650521 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650524 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650528 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650531 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650535 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650538 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650541 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650544 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650547 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650550 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650553 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650556 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650559 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650562 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650565 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650568 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:32:36.655960 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650571 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650574 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650577 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650580 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650583 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650586 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650589 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650592 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650595 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650598 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650601 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650603 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650606 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650609 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650613 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650617 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650620 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650623 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650626 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:32:36.656490 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650629 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650632 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650635 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650638 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650640 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650643 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650646 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650649 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650651 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650654 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650657 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650660 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650662 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650665 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650668 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650670 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650673 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650676 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650679 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:32:36.657026 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650682 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650686 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650690 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650694 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650697 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650700 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650703 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:36.650706 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.650712 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.650847 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.655282 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.656789 2559 server.go:1019] "Starting client certificate rotation" Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.656902 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 14:32:36.657638 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.656949 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 14:32:36.660859 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.660844 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 14:32:36.662130 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.662108 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 14:32:36.675527 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.675505 2559 log.go:25] "Validated CRI v1 runtime API" Jul 21 14:32:36.677878 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.677863 2559 log.go:25] "Validated CRI v1 image API" Jul 21 14:32:36.679278 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.679189 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 14:32:36.682772 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.682745 2559 fs.go:135] Filesystem UUIDs: map[43d97bd8-59e6-432b-9186-182141442760:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c516f98e-ee41-4152-b85d-7e135f869a51:/dev/nvme0n1p3] Jul 21 14:32:36.682879 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.682771 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 14:32:36.688497 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.688358 2559 manager.go:216] Machine: {Timestamp:2026-07-21 14:32:36.687817311 +0000 UTC m=+1.176008509 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100549 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec262070cba120f0cf067d946d98d9fc SystemUUID:ec262070-cba1-20f0-cf06-7d946d98d9fc BootID:e047ac08-a855-4102-9fa6-5c0fa21a1347 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:ce:b6:16:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:ce:b6:16:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:98:58:33:ad:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 14:32:36.688497 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.688481 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 14:32:36.688688 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.688606 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 14:32:36.692945 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.692914 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 14:32:36.693132 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.692953 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-112.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 14:32:36.693769 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.693757 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 14:32:36.693827 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.693775 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 14:32:36.693827 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.693786 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 14:32:36.694276 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.694257 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 14:32:36.695378 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.695362 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 14:32:36.696758 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.696743 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 14:32:36.700267 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700251 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 21 14:32:36.700445 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700432 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 14:32:36.700509 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700473 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 21 14:32:36.700509 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700488 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 14:32:36.700602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700509 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 14:32:36.700602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700533 2559 kubelet.go:398] "Adding apiserver pod source" Jul 21 14:32:36.700602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700547 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 14:32:36.700991 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.700976 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 14:32:36.701057 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.701003 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 14:32:36.703290 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.703275 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 14:32:36.704809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.704794 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 14:32:36.704865 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.704828 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 14:32:36.705522 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705511 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 14:32:36.705566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705527 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 14:32:36.705566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705534 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 14:32:36.705566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705540 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 14:32:36.705566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705550 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 14:32:36.705566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705564 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 14:32:36.705701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705574 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 14:32:36.705701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705581 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 14:32:36.705701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705588 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 14:32:36.705701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705594 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 14:32:36.705701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705609 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 14:32:36.705701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.705616 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 14:32:36.706463 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.706452 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 14:32:36.706463 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.706462 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 14:32:36.709966 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.709950 2559 server.go:1268] "Started kubelet" Jul 21 14:32:36.710595 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.710078 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 14:32:36.710595 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.710192 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 14:32:36.710723 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.710580 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 14:32:36.711113 ip-10-0-130-112 systemd[1]: Started Kubernetes Kubelet. Jul 21 14:32:36.711770 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.711747 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 14:32:36.713285 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.713262 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 21 14:32:36.715058 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.715035 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-112.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 14:32:36.715058 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.715042 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 14:32:36.715350 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.715042 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 14:32:36.717084 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.717062 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 14:32:36.720310 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.715060 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-112.ec2.internal.18c454964419a9cc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-112.ec2.internal,UID:ip-10-0-130-112.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-112.ec2.internal,},FirstTimestamp:2026-07-21 14:32:36.70991918 +0000 UTC m=+1.198110377,LastTimestamp:2026-07-21 14:32:36.70991918 +0000 UTC m=+1.198110377,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-112.ec2.internal,}" Jul 21 14:32:36.720419 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.720314 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 14:32:36.720419 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.720348 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 14:32:36.720774 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.720750 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:36.721019 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.721001 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 14:32:36.721110 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.721085 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 14:32:36.721419 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.721401 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 14:32:36.721571 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.721554 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 21 14:32:36.721700 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.721673 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 21 14:32:36.722001 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.721985 2559 factory.go:153] Registering CRI-O factory Jul 21 14:32:36.722082 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.722005 2559 factory.go:223] Registration of the crio container factory successfully Jul 21 14:32:36.722082 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.722066 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 14:32:36.722082 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.722077 2559 factory.go:55] Registering systemd factory Jul 21 14:32:36.722082 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.722083 2559 factory.go:223] Registration of the systemd container factory successfully Jul 21 14:32:36.722259 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.722108 2559 factory.go:103] Registering Raw factory Jul 21 14:32:36.722259 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.722122 2559 manager.go:1196] Started watching for new ooms in manager Jul 21 14:32:36.722769 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.722581 2559 manager.go:349] Starting recovery of all containers Jul 21 14:32:36.726735 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.726708 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2r6w2" Jul 21 14:32:36.732746 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.732686 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 14:32:36.733010 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.732868 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 14:32:36.733706 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.733669 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2r6w2" Jul 21 14:32:36.734221 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.734141 2559 manager.go:354] Recovery completed Jul 21 14:32:36.741144 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.741130 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:32:36.743078 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.743063 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:32:36.743137 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.743092 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:32:36.743137 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.743103 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:32:36.743704 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.743690 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 14:32:36.743704 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.743703 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 14:32:36.743796 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.743718 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 21 14:32:36.743796 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.743694 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:32:36.744533 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.744515 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:32:36.744605 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.744545 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:32:36.744605 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.744571 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:32:36.744983 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.744971 2559 policy_none.go:49] "None policy: Start" Jul 21 14:32:36.745015 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.744988 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 14:32:36.745015 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.744999 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 14:32:36.745460 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.745391 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-112.ec2.internal.18c454964613a03b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-112.ec2.internal,UID:ip-10-0-130-112.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-112.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-112.ec2.internal,},FirstTimestamp:2026-07-21 14:32:36.743077947 +0000 UTC m=+1.231269148,LastTimestamp:2026-07-21 14:32:36.743077947 +0000 UTC m=+1.231269148,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-112.ec2.internal,}" Jul 21 14:32:36.746133 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.746119 2559 policy_none.go:47] "Start" Jul 21 14:32:36.796077 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.795909 2559 manager.go:337] "Starting Device Plugin manager" Jul 21 14:32:36.796197 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.796102 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 14:32:36.796197 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.796120 2559 server.go:87] "Starting device plugin registration server" Jul 21 14:32:36.796435 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.796423 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 14:32:36.796467 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.796437 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 14:32:36.796698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.796533 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 14:32:36.796698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.796645 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 14:32:36.796698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.796653 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 14:32:36.797343 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.797316 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 14:32:36.797438 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.797381 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:36.902248 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.902184 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:32:36.905563 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.905546 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:32:36.905669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.905578 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:32:36.905669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.905589 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:32:36.905669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.905618 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-112.ec2.internal" Jul 21 14:32:36.908081 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.908054 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 14:32:36.909427 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.909407 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 14:32:36.909540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.909444 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 14:32:36.909540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.909463 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 14:32:36.909540 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.909495 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 14:32:36.914615 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.914587 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:32:36.915587 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:36.915571 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-112.ec2.internal" Jul 21 14:32:36.915663 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.915601 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-112.ec2.internal\": node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:36.938744 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:36.938718 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.010603 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.010504 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal"] Jul 21 14:32:37.010603 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.010594 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:32:37.011662 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.011644 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:32:37.011760 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.011679 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:32:37.011760 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.011693 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:32:37.013194 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.013177 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:32:37.013363 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.013347 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.013422 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.013379 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:32:37.013994 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.013975 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:32:37.013994 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.013985 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:32:37.014158 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.014007 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:32:37.014158 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.014008 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:32:37.014158 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.014022 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:32:37.014158 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.014028 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:32:37.015439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.015426 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.015494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.015449 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:32:37.016184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.016168 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:32:37.016293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.016194 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:32:37.016293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.016209 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:32:37.022679 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.022659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1d033722713bf948e6f973d6228bc956-config\") pod \"kube-apiserver-proxy-ip-10-0-130-112.ec2.internal\" (UID: \"1d033722713bf948e6f973d6228bc956\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.022772 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.022693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27e815f37a7bba8db1a0a898d0836892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"27e815f37a7bba8db1a0a898d0836892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.022772 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.022722 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e815f37a7bba8db1a0a898d0836892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"27e815f37a7bba8db1a0a898d0836892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.038814 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.038791 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.040188 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.040172 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-112.ec2.internal\" not found" node="ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.044793 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.044775 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-112.ec2.internal\" not found" node="ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.123020 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.122989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1d033722713bf948e6f973d6228bc956-config\") pod \"kube-apiserver-proxy-ip-10-0-130-112.ec2.internal\" (UID: \"1d033722713bf948e6f973d6228bc956\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.123020 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.123019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27e815f37a7bba8db1a0a898d0836892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"27e815f37a7bba8db1a0a898d0836892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.123293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.123036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e815f37a7bba8db1a0a898d0836892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"27e815f37a7bba8db1a0a898d0836892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.123293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.123084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e815f37a7bba8db1a0a898d0836892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"27e815f37a7bba8db1a0a898d0836892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.123293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.123111 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1d033722713bf948e6f973d6228bc956-config\") pod \"kube-apiserver-proxy-ip-10-0-130-112.ec2.internal\" (UID: \"1d033722713bf948e6f973d6228bc956\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.123293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.123120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27e815f37a7bba8db1a0a898d0836892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal\" (UID: \"27e815f37a7bba8db1a0a898d0836892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.140019 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.139979 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.240633 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.240582 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.341600 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.341570 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.342767 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.342753 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.349237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.349200 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" Jul 21 14:32:37.442385 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.442340 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.543550 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.543511 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.642108 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.642047 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:32:37.644320 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.644301 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.656342 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.656310 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 14:32:37.656462 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.656441 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 14:32:37.656506 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.656476 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 14:32:37.738323 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.738290 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 14:27:37 +0000 UTC" deadline="2028-04-21 09:14:35.59117223 +0000 UTC" Jul 21 14:32:37.738323 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:37.738315 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15354h41m57.852860214s" Jul 21 14:32:37.744851 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.744826 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.845309 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.845262 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:37.945524 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:37.945443 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:38.046029 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.046002 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-112.ec2.internal\" not found" Jul 21 14:32:38.134058 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.134026 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:32:38.220326 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.220216 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:32:38.221291 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.221274 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" Jul 21 14:32:38.233312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.233287 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 14:32:38.235856 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.235840 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" Jul 21 14:32:38.244769 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.244747 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 14:32:38.625973 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:38.625925 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d033722713bf948e6f973d6228bc956.slice/crio-de12d6f78995ca6615690606d31059e99b79a5ac5a44ab02a06ac7586cc6c3e7 WatchSource:0}: Error finding container de12d6f78995ca6615690606d31059e99b79a5ac5a44ab02a06ac7586cc6c3e7: Status 404 returned error can't find the container with id de12d6f78995ca6615690606d31059e99b79a5ac5a44ab02a06ac7586cc6c3e7 Jul 21 14:32:38.631762 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.631747 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:32:38.704328 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.704285 2559 apiserver.go:52] "Watching apiserver" Jul 21 14:32:38.712799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.712774 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 14:32:38.713154 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.713133 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-8wkxm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal","openshift-multus/multus-additional-cni-plugins-hpksb","openshift-multus/multus-sdfsq","openshift-multus/network-metrics-daemon-crw6b","kube-system/global-pull-secret-syncer-wpdjf","openshift-cluster-node-tuning-operator/tuned-q6c9v","openshift-image-registry/node-ca-9mjkx","openshift-network-diagnostics/network-check-target-wlfd9","openshift-network-operator/iptables-alerter-snbnj","openshift-ovn-kubernetes/ovnkube-node-nbwvc","kube-system/konnectivity-agent-lc8kv","kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8"] Jul 21 14:32:38.717029 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.717013 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.720506 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.719763 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:32:38.720506 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.720027 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 14:32:38.720506 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.720169 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.720970 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.720946 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zgbbv\"" Jul 21 14:32:38.722334 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.722182 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 14:32:38.723495 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.723473 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.724254 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.724041 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 14:32:38.724661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.724391 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 14:32:38.724661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.724496 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 14:32:38.724661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.724534 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jvq4p\"" Jul 21 14:32:38.724661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.724534 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 14:32:38.724956 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.724679 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 14:32:38.724956 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.724749 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 14:32:38.726399 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.726378 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.726597 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.726576 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 14:32:38.726761 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.726745 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z4bf8\"" Jul 21 14:32:38.726821 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.726762 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 14:32:38.727002 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.726989 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 14:32:38.727071 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.727053 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 14:32:38.728215 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.728189 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 14:32:38.728736 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.728717 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 14:32:38.728873 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.728856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:38.728957 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.728936 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:38.729079 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.729064 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nnkkc\"" Jul 21 14:32:38.731160 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.731138 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.731256 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.731212 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:38.733549 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.733529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:38.733621 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.733596 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:38.733940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.733925 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 14:32:38.735915 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.735898 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.736553 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-kubernetes\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.736635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736565 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-tuned\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.736635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd1b571-fb06-4715-8e64-96e42a20a05d-tmp\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.736635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736618 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-systemd-units\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.736802 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-slash\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.736802 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-ovn\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.736802 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736737 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-node-log\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.736802 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-cni-bin\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.736987 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-var-lib-kubelet\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.736987 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysctl-d\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.736987 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-run\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.736987 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-sys\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.736987 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxzlh\" (UniqueName: \"kubernetes.io/projected/dfd1b571-fb06-4715-8e64-96e42a20a05d-kube-api-access-fxzlh\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.737148 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.736999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-etc-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737148 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737031 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737148 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysctl-conf\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.737148 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovnkube-config\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovnkube-script-lib\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737179 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cnibin\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.737302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.737398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.737398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737344 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vlx\" (UniqueName: \"kubernetes.io/projected/34fbe162-4fcd-4382-9b5d-eb2873fe641c-kube-api-access-x4vlx\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.737398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737370 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-modprobe-d\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.737398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-lib-modules\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.737525 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-var-lib-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737525 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737525 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.737525 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737506 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysconfig\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.737643 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-host\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.737643 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737578 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gkn\" (UniqueName: \"kubernetes.io/projected/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-kube-api-access-h5gkn\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737643 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-system-cni-dir\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.737643 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-kubelet\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-run-netns\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737712 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-log-socket\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-cni-netd\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-env-overrides\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovn-node-metrics-cert\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-os-release\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.737809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-systemd\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.738075 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-systemd\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.738075 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.737852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.738437 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.738418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.738799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.738735 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 14:32:38.738799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.738756 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 14:32:38.738799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.738772 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n45rr\"" Jul 21 14:32:38.739035 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.738801 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 14:32:38.741090 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.741058 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.741187 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.741109 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 14:32:38.741307 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.741271 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 14:32:38.741573 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.741554 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 14:32:38.741677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.741658 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-87lhf\"" Jul 21 14:32:38.743676 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.743655 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.745849 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.745833 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:32:38.745920 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.745851 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 14:32:38.747049 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.746324 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 14:32:38.747049 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.746426 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:38.747049 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.746824 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 14:32:38.749498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.747253 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 14:32:38.749498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.747529 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wwbl\"" Jul 21 14:32:38.749498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.747549 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dgwn4\"" Jul 21 14:32:38.749498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.748908 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 14:32:38.749498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.749167 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 14:32:38.750208 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.750149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s7l79\"" Jul 21 14:32:38.764489 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.764468 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l6f8c" Jul 21 14:32:38.773437 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.773420 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l6f8c" Jul 21 14:32:38.800226 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.800191 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:32:38.822407 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.822381 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 14:32:38.838446 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-systemd\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.838598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96be4230-5b49-4819-b027-1c57fc60964d-host\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.838598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db2d40c8-02bb-461d-b2da-77af6c63dc7a-cni-binary-copy\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.838598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-conf-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.838598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92704e7d-742e-4358-a920-286027d064b0-kubelet-config\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.838598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-systemd-units\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-slash\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838590 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-systemd\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-node-log\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-node-log\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-cni-bin\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-slash\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-systemd-units\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838664 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-os-release\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-cni-bin\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysctl-d\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-run\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-sys\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-etc-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovnkube-script-lib\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838796 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-sys\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cnibin\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-etc-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838836 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-run\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76nx\" (UniqueName: \"kubernetes.io/projected/abb41b77-8b49-4565-817b-76b89de2ea8a-kube-api-access-k76nx\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.838928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysctl-d\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-daemon-config\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838881 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cnibin\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-etc-kubernetes\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2hf\" (UniqueName: \"kubernetes.io/projected/070e8201-fcbc-45cc-b266-1b3625351d2d-kube-api-access-xs2hf\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e9595f3-e87e-46b8-993b-9fb4981d50d8-hosts-file\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.838985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-cni-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-cni-multus\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-host\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-host\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839083 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839126 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/420a38b5-47bb-4326-927f-cc0a80bde4c4-agent-certs\") pod \"konnectivity-agent-lc8kv\" (UID: \"420a38b5-47bb-4326-927f-cc0a80bde4c4\") " pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:38.839842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gkn\" (UniqueName: \"kubernetes.io/projected/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-kube-api-access-h5gkn\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-system-cni-dir\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839361 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-system-cni-dir\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovnkube-script-lib\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92704e7d-742e-4358-a920-286027d064b0-dbus\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839425 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-cni-netd\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-env-overrides\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96be4230-5b49-4819-b027-1c57fc60964d-serviceca\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-cni-netd\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-cnibin\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-socket-dir-parent\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjtb\" (UniqueName: \"kubernetes.io/projected/41713217-eaa0-4eb2-b6f0-d89170d86af0-kube-api-access-8wjtb\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/420a38b5-47bb-4326-927f-cc0a80bde4c4-konnectivity-ca\") pod \"konnectivity-agent-lc8kv\" (UID: \"420a38b5-47bb-4326-927f-cc0a80bde4c4\") " pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76l5j\" (UniqueName: \"kubernetes.io/projected/96be4230-5b49-4819-b027-1c57fc60964d-kube-api-access-76l5j\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-systemd\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-systemd\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.840726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839676 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9595f3-e87e-46b8-993b-9fb4981d50d8-tmp-dir\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-kubernetes\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839737 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-tuned\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd1b571-fb06-4715-8e64-96e42a20a05d-tmp\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-ovn\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-kubernetes\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839813 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-socket-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-sys-fs\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839839 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-ovn\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxcg\" (UniqueName: \"kubernetes.io/projected/2e9595f3-e87e-46b8-993b-9fb4981d50d8-kube-api-access-fpxcg\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-var-lib-kubelet\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.839988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxzlh\" (UniqueName: \"kubernetes.io/projected/dfd1b571-fb06-4715-8e64-96e42a20a05d-kube-api-access-fxzlh\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840008 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840041 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.841557 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840060 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-var-lib-kubelet\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovnkube-config\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vlx\" (UniqueName: \"kubernetes.io/projected/34fbe162-4fcd-4382-9b5d-eb2873fe641c-kube-api-access-x4vlx\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-system-cni-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysctl-conf\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840213 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-kubelet\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysctl-conf\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840469 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-multus-certs\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070e8201-fcbc-45cc-b266-1b3625351d2d-host-slash\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-modprobe-d\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-lib-modules\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-var-lib-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwtt\" (UniqueName: \"kubernetes.io/projected/db2d40c8-02bb-461d-b2da-77af6c63dc7a-kube-api-access-hfwtt\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysconfig\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-k8s-cni-cncf-io\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-modprobe-d\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840713 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-lib-modules\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-netns\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-env-overrides\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-hostroot\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovnkube-config\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-var-lib-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-sysconfig\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-cni-bin\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/070e8201-fcbc-45cc-b266-1b3625351d2d-iptables-alerter-script\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-kubelet\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-run-netns\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840959 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.842602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-kubelet\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840984 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-log-socket\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.840987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-host-run-netns\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovn-node-metrics-cert\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-log-socket\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-os-release\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-run-openvswitch\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-registration-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841085 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-device-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34fbe162-4fcd-4382-9b5d-eb2873fe641c-os-release\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.843100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.841173 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34fbe162-4fcd-4382-9b5d-eb2873fe641c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.843566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.843468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dfd1b571-fb06-4715-8e64-96e42a20a05d-etc-tuned\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.843566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.843520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd1b571-fb06-4715-8e64-96e42a20a05d-tmp\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.843641 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.843625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-ovn-node-metrics-cert\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.847478 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.847445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gkn\" (UniqueName: \"kubernetes.io/projected/e5287c9f-3012-4c0b-bed8-3cd0af5c365b-kube-api-access-h5gkn\") pod \"ovnkube-node-nbwvc\" (UID: \"e5287c9f-3012-4c0b-bed8-3cd0af5c365b\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:38.847574 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.847474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxzlh\" (UniqueName: \"kubernetes.io/projected/dfd1b571-fb06-4715-8e64-96e42a20a05d-kube-api-access-fxzlh\") pod \"tuned-q6c9v\" (UID: \"dfd1b571-fb06-4715-8e64-96e42a20a05d\") " pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:38.848080 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.848055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vlx\" (UniqueName: \"kubernetes.io/projected/34fbe162-4fcd-4382-9b5d-eb2873fe641c-kube-api-access-x4vlx\") pod \"multus-additional-cni-plugins-hpksb\" (UID: \"34fbe162-4fcd-4382-9b5d-eb2873fe641c\") " pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:38.916762 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.916709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" event={"ID":"1d033722713bf948e6f973d6228bc956","Type":"ContainerStarted","Data":"de12d6f78995ca6615690606d31059e99b79a5ac5a44ab02a06ac7586cc6c3e7"} Jul 21 14:32:38.941656 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941622 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-system-cni-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941656 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-kubelet\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-multus-certs\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070e8201-fcbc-45cc-b266-1b3625351d2d-host-slash\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwtt\" (UniqueName: \"kubernetes.io/projected/db2d40c8-02bb-461d-b2da-77af6c63dc7a-kube-api-access-hfwtt\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-k8s-cni-cncf-io\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-kubelet\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941775 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-multus-certs\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-k8s-cni-cncf-io\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-system-cni-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-netns\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941784 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/070e8201-fcbc-45cc-b266-1b3625351d2d-host-slash\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941830 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-run-netns\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941865 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-hostroot\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941890 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-hostroot\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941911 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-cni-bin\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.941943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941945 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941956 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-cni-bin\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.941974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/070e8201-fcbc-45cc-b266-1b3625351d2d-iptables-alerter-script\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942005 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-registration-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-device-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96be4230-5b49-4819-b027-1c57fc60964d-host\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.942076 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-registration-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-device-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.942174 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret podName:92704e7d-742e-4358-a920-286027d064b0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:39.442134482 +0000 UTC m=+3.930325673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret") pod "global-pull-secret-syncer-wpdjf" (UID: "92704e7d-742e-4358-a920-286027d064b0") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942081 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db2d40c8-02bb-461d-b2da-77af6c63dc7a-cni-binary-copy\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96be4230-5b49-4819-b027-1c57fc60964d-host\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-conf-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-conf-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92704e7d-742e-4358-a920-286027d064b0-kubelet-config\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-os-release\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76nx\" (UniqueName: \"kubernetes.io/projected/abb41b77-8b49-4565-817b-76b89de2ea8a-kube-api-access-k76nx\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.942661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-os-release\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942367 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92704e7d-742e-4358-a920-286027d064b0-kubelet-config\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-daemon-config\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-etc-kubernetes\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2hf\" (UniqueName: \"kubernetes.io/projected/070e8201-fcbc-45cc-b266-1b3625351d2d-kube-api-access-xs2hf\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942644 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-etc-kubernetes\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/070e8201-fcbc-45cc-b266-1b3625351d2d-iptables-alerter-script\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942641 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e9595f3-e87e-46b8-993b-9fb4981d50d8-hosts-file\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-cni-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-cni-multus\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e9595f3-e87e-46b8-993b-9fb4981d50d8-hosts-file\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-host-var-lib-cni-multus\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db2d40c8-02bb-461d-b2da-77af6c63dc7a-cni-binary-copy\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/420a38b5-47bb-4326-927f-cc0a80bde4c4-agent-certs\") pod \"konnectivity-agent-lc8kv\" (UID: \"420a38b5-47bb-4326-927f-cc0a80bde4c4\") " pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.942839 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:38.943540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942861 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942874 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-cni-dir\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92704e7d-742e-4358-a920-286027d064b0-dbus\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.942905 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs podName:41713217-eaa0-4eb2-b6f0-d89170d86af0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:39.442887126 +0000 UTC m=+3.931078349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs") pod "network-metrics-daemon-crw6b" (UID: "41713217-eaa0-4eb2-b6f0-d89170d86af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92704e7d-742e-4358-a920-286027d064b0-dbus\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-daemon-config\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96be4230-5b49-4819-b027-1c57fc60964d-serviceca\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.942984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-cnibin\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-socket-dir-parent\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjtb\" (UniqueName: \"kubernetes.io/projected/41713217-eaa0-4eb2-b6f0-d89170d86af0-kube-api-access-8wjtb\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-cnibin\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/420a38b5-47bb-4326-927f-cc0a80bde4c4-konnectivity-ca\") pod \"konnectivity-agent-lc8kv\" (UID: \"420a38b5-47bb-4326-927f-cc0a80bde4c4\") " pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db2d40c8-02bb-461d-b2da-77af6c63dc7a-multus-socket-dir-parent\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76l5j\" (UniqueName: \"kubernetes.io/projected/96be4230-5b49-4819-b027-1c57fc60964d-kube-api-access-76l5j\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9595f3-e87e-46b8-993b-9fb4981d50d8-tmp-dir\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.944181 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-socket-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.944799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-sys-fs\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.944799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96be4230-5b49-4819-b027-1c57fc60964d-serviceca\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:38.944799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxcg\" (UniqueName: \"kubernetes.io/projected/2e9595f3-e87e-46b8-993b-9fb4981d50d8-kube-api-access-fpxcg\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.944799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-sys-fs\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.944799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abb41b77-8b49-4565-817b-76b89de2ea8a-socket-dir\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.944799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9595f3-e87e-46b8-993b-9fb4981d50d8-tmp-dir\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.944799 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.943885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/420a38b5-47bb-4326-927f-cc0a80bde4c4-konnectivity-ca\") pod \"konnectivity-agent-lc8kv\" (UID: \"420a38b5-47bb-4326-927f-cc0a80bde4c4\") " pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:38.945404 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.945386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/420a38b5-47bb-4326-927f-cc0a80bde4c4-agent-certs\") pod \"konnectivity-agent-lc8kv\" (UID: \"420a38b5-47bb-4326-927f-cc0a80bde4c4\") " pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:38.951763 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.951739 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:32:38.951763 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.951765 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:32:38.951939 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.951782 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nrdpg for pod openshift-network-diagnostics/network-check-target-wlfd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:38.951939 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:38.951873 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg podName:74174e45-7725-4b5f-91d1-43ec79896c38 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:39.451832851 +0000 UTC m=+3.940024037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nrdpg" (UniqueName: "kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg") pod "network-check-target-wlfd9" (UID: "74174e45-7725-4b5f-91d1-43ec79896c38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:38.953666 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.953633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwtt\" (UniqueName: \"kubernetes.io/projected/db2d40c8-02bb-461d-b2da-77af6c63dc7a-kube-api-access-hfwtt\") pod \"multus-sdfsq\" (UID: \"db2d40c8-02bb-461d-b2da-77af6c63dc7a\") " pod="openshift-multus/multus-sdfsq" Jul 21 14:32:38.953777 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.953667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2hf\" (UniqueName: \"kubernetes.io/projected/070e8201-fcbc-45cc-b266-1b3625351d2d-kube-api-access-xs2hf\") pod \"iptables-alerter-snbnj\" (UID: \"070e8201-fcbc-45cc-b266-1b3625351d2d\") " pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:38.953841 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.953786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76nx\" (UniqueName: \"kubernetes.io/projected/abb41b77-8b49-4565-817b-76b89de2ea8a-kube-api-access-k76nx\") pod \"aws-ebs-csi-driver-node-q5gm8\" (UID: \"abb41b77-8b49-4565-817b-76b89de2ea8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:38.954205 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.954184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjtb\" (UniqueName: \"kubernetes.io/projected/41713217-eaa0-4eb2-b6f0-d89170d86af0-kube-api-access-8wjtb\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:38.954534 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.954518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxcg\" (UniqueName: \"kubernetes.io/projected/2e9595f3-e87e-46b8-993b-9fb4981d50d8-kube-api-access-fpxcg\") pod \"node-resolver-8wkxm\" (UID: \"2e9595f3-e87e-46b8-993b-9fb4981d50d8\") " pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:38.954703 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:38.954687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76l5j\" (UniqueName: \"kubernetes.io/projected/96be4230-5b49-4819-b027-1c57fc60964d-kube-api-access-76l5j\") pod \"node-ca-9mjkx\" (UID: \"96be4230-5b49-4819-b027-1c57fc60964d\") " pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:39.029130 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.029047 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" Jul 21 14:32:39.036011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.035987 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:32:39.039250 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.039215 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd1b571_fb06_4715_8e64_96e42a20a05d.slice/crio-b2d1f9b362e08efeadf9b5d8a67bed45cb8dc0dc0b929a1d0eaa45d5ee3e1f63 WatchSource:0}: Error finding container b2d1f9b362e08efeadf9b5d8a67bed45cb8dc0dc0b929a1d0eaa45d5ee3e1f63: Status 404 returned error can't find the container with id b2d1f9b362e08efeadf9b5d8a67bed45cb8dc0dc0b929a1d0eaa45d5ee3e1f63 Jul 21 14:32:39.044190 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.044167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hpksb" Jul 21 14:32:39.044876 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.044851 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5287c9f_3012_4c0b_bed8_3cd0af5c365b.slice/crio-08ca5971ddc2f5092071a9547ffc5bf2f6a14135563308e06c9a1a14053cf606 WatchSource:0}: Error finding container 08ca5971ddc2f5092071a9547ffc5bf2f6a14135563308e06c9a1a14053cf606: Status 404 returned error can't find the container with id 08ca5971ddc2f5092071a9547ffc5bf2f6a14135563308e06c9a1a14053cf606 Jul 21 14:32:39.049066 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.049050 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sdfsq" Jul 21 14:32:39.053631 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.053606 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fbe162_4fcd_4382_9b5d_eb2873fe641c.slice/crio-083e3bbb28d54d188be95d8fe40ec66a5b70a7c3889d67cab06fdd8cd3943845 WatchSource:0}: Error finding container 083e3bbb28d54d188be95d8fe40ec66a5b70a7c3889d67cab06fdd8cd3943845: Status 404 returned error can't find the container with id 083e3bbb28d54d188be95d8fe40ec66a5b70a7c3889d67cab06fdd8cd3943845 Jul 21 14:32:39.055318 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.055299 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" Jul 21 14:32:39.060214 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.060174 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2d40c8_02bb_461d_b2da_77af6c63dc7a.slice/crio-7505176aa97c01f320aa558c7c58a3b7ee82fe54f4221d47e0a1b07bdfe933d3 WatchSource:0}: Error finding container 7505176aa97c01f320aa558c7c58a3b7ee82fe54f4221d47e0a1b07bdfe933d3: Status 404 returned error can't find the container with id 7505176aa97c01f320aa558c7c58a3b7ee82fe54f4221d47e0a1b07bdfe933d3 Jul 21 14:32:39.061162 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.061003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9mjkx" Jul 21 14:32:39.067922 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.067902 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-snbnj" Jul 21 14:32:39.072541 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.072522 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb41b77_8b49_4565_817b_76b89de2ea8a.slice/crio-94d35f9dbb5b185b73ea46c2a9ccc1c8f29a1b840ecbd947a7d6deab86bb1bb8 WatchSource:0}: Error finding container 94d35f9dbb5b185b73ea46c2a9ccc1c8f29a1b840ecbd947a7d6deab86bb1bb8: Status 404 returned error can't find the container with id 94d35f9dbb5b185b73ea46c2a9ccc1c8f29a1b840ecbd947a7d6deab86bb1bb8 Jul 21 14:32:39.073317 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.073299 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wkxm" Jul 21 14:32:39.079314 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.079287 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:32:39.086446 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.086419 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070e8201_fcbc_45cc_b266_1b3625351d2d.slice/crio-a3ed70e0e3fd1f38fa68e11bcba7a0f0531d4a5687e49681f1e7cc095b061a97 WatchSource:0}: Error finding container a3ed70e0e3fd1f38fa68e11bcba7a0f0531d4a5687e49681f1e7cc095b061a97: Status 404 returned error can't find the container with id a3ed70e0e3fd1f38fa68e11bcba7a0f0531d4a5687e49681f1e7cc095b061a97 Jul 21 14:32:39.087948 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.087850 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96be4230_5b49_4819_b027_1c57fc60964d.slice/crio-70f06f2f3745330768c92ddc9246bb4798f293d0eac546ef72657ea6feb3d818 WatchSource:0}: Error finding container 70f06f2f3745330768c92ddc9246bb4798f293d0eac546ef72657ea6feb3d818: Status 404 returned error can't find the container with id 70f06f2f3745330768c92ddc9246bb4798f293d0eac546ef72657ea6feb3d818 Jul 21 14:32:39.090852 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.090818 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e815f37a7bba8db1a0a898d0836892.slice/crio-c4b6298560499cfb7c1533d1be821f8cd4f2b497668dc6f009d3266176e3d04b WatchSource:0}: Error finding container c4b6298560499cfb7c1533d1be821f8cd4f2b497668dc6f009d3266176e3d04b: Status 404 returned error can't find the container with id c4b6298560499cfb7c1533d1be821f8cd4f2b497668dc6f009d3266176e3d04b Jul 21 14:32:39.091659 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.091534 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9595f3_e87e_46b8_993b_9fb4981d50d8.slice/crio-3fa3040261c7a3b6c25a3bf957fff411963c41cb291d6549a43e1c6759953abd WatchSource:0}: Error finding container 3fa3040261c7a3b6c25a3bf957fff411963c41cb291d6549a43e1c6759953abd: Status 404 returned error can't find the container with id 3fa3040261c7a3b6c25a3bf957fff411963c41cb291d6549a43e1c6759953abd Jul 21 14:32:39.095790 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:32:39.095765 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420a38b5_47bb_4326_927f_cc0a80bde4c4.slice/crio-522d9794278039a635b5fb22175ba5898b1945f28cf99c209ea4507156b516d0 WatchSource:0}: Error finding container 522d9794278039a635b5fb22175ba5898b1945f28cf99c209ea4507156b516d0: Status 404 returned error can't find the container with id 522d9794278039a635b5fb22175ba5898b1945f28cf99c209ea4507156b516d0 Jul 21 14:32:39.448470 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.448429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:39.448664 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.448496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:39.448664 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.448627 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:39.448756 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.448693 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:39.448756 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.448717 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret podName:92704e7d-742e-4358-a920-286027d064b0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:40.448693074 +0000 UTC m=+4.936884261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret") pod "global-pull-secret-syncer-wpdjf" (UID: "92704e7d-742e-4358-a920-286027d064b0") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:39.448756 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.448752 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs podName:41713217-eaa0-4eb2-b6f0-d89170d86af0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:40.448738458 +0000 UTC m=+4.936929642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs") pod "network-metrics-daemon-crw6b" (UID: "41713217-eaa0-4eb2-b6f0-d89170d86af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:39.549856 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.549757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:39.550046 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.549945 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:32:39.550046 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.549976 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:32:39.550046 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.549992 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nrdpg for pod openshift-network-diagnostics/network-check-target-wlfd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:39.550226 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.550071 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg podName:74174e45-7725-4b5f-91d1-43ec79896c38 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:40.550051495 +0000 UTC m=+5.038242685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrdpg" (UniqueName: "kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg") pod "network-check-target-wlfd9" (UID: "74174e45-7725-4b5f-91d1-43ec79896c38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:39.776972 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.776828 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 14:27:39 +0000 UTC" deadline="2028-04-08 16:39:06.655864118 +0000 UTC" Jul 21 14:32:39.776972 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.776922 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15050h6m26.878947069s" Jul 21 14:32:39.910609 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.910391 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:39.910609 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:39.910554 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:39.927430 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.927394 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sdfsq" event={"ID":"db2d40c8-02bb-461d-b2da-77af6c63dc7a","Type":"ContainerStarted","Data":"7505176aa97c01f320aa558c7c58a3b7ee82fe54f4221d47e0a1b07bdfe933d3"} Jul 21 14:32:39.931702 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.931588 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerStarted","Data":"083e3bbb28d54d188be95d8fe40ec66a5b70a7c3889d67cab06fdd8cd3943845"} Jul 21 14:32:39.939509 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.939476 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" event={"ID":"dfd1b571-fb06-4715-8e64-96e42a20a05d","Type":"ContainerStarted","Data":"b2d1f9b362e08efeadf9b5d8a67bed45cb8dc0dc0b929a1d0eaa45d5ee3e1f63"} Jul 21 14:32:39.942196 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.942169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wkxm" event={"ID":"2e9595f3-e87e-46b8-993b-9fb4981d50d8","Type":"ContainerStarted","Data":"3fa3040261c7a3b6c25a3bf957fff411963c41cb291d6549a43e1c6759953abd"} Jul 21 14:32:39.950712 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.950682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" event={"ID":"abb41b77-8b49-4565-817b-76b89de2ea8a","Type":"ContainerStarted","Data":"94d35f9dbb5b185b73ea46c2a9ccc1c8f29a1b840ecbd947a7d6deab86bb1bb8"} Jul 21 14:32:39.960237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.960113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"08ca5971ddc2f5092071a9547ffc5bf2f6a14135563308e06c9a1a14053cf606"} Jul 21 14:32:39.964836 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.964805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lc8kv" event={"ID":"420a38b5-47bb-4326-927f-cc0a80bde4c4","Type":"ContainerStarted","Data":"522d9794278039a635b5fb22175ba5898b1945f28cf99c209ea4507156b516d0"} Jul 21 14:32:39.976955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.976867 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" event={"ID":"27e815f37a7bba8db1a0a898d0836892","Type":"ContainerStarted","Data":"c4b6298560499cfb7c1533d1be821f8cd4f2b497668dc6f009d3266176e3d04b"} Jul 21 14:32:39.983521 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.983490 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9mjkx" event={"ID":"96be4230-5b49-4819-b027-1c57fc60964d","Type":"ContainerStarted","Data":"70f06f2f3745330768c92ddc9246bb4798f293d0eac546ef72657ea6feb3d818"} Jul 21 14:32:39.989413 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:39.989343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-snbnj" event={"ID":"070e8201-fcbc-45cc-b266-1b3625351d2d","Type":"ContainerStarted","Data":"a3ed70e0e3fd1f38fa68e11bcba7a0f0531d4a5687e49681f1e7cc095b061a97"} Jul 21 14:32:40.459295 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.459195 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:40.459295 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.459272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:40.459528 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.459426 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:40.459528 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.459492 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs podName:41713217-eaa0-4eb2-b6f0-d89170d86af0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:42.459471429 +0000 UTC m=+6.947662615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs") pod "network-metrics-daemon-crw6b" (UID: "41713217-eaa0-4eb2-b6f0-d89170d86af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:40.459936 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.459909 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:40.460040 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.459975 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret podName:92704e7d-742e-4358-a920-286027d064b0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:42.459959377 +0000 UTC m=+6.948150561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret") pod "global-pull-secret-syncer-wpdjf" (UID: "92704e7d-742e-4358-a920-286027d064b0") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:40.531534 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.531499 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:32:40.559796 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.559759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:40.559996 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.559956 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:32:40.559996 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.559977 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:32:40.559996 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.559991 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nrdpg for pod openshift-network-diagnostics/network-check-target-wlfd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:40.560172 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.560054 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg podName:74174e45-7725-4b5f-91d1-43ec79896c38 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:42.560035167 +0000 UTC m=+7.048226355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrdpg" (UniqueName: "kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg") pod "network-check-target-wlfd9" (UID: "74174e45-7725-4b5f-91d1-43ec79896c38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:40.777498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.777402 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 14:27:39 +0000 UTC" deadline="2028-02-26 18:39:46.008174296 +0000 UTC" Jul 21 14:32:40.777498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.777447 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14044h7m5.230732582s" Jul 21 14:32:40.910818 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.910776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:40.911073 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.910938 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:40.911651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:40.911469 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:40.911651 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:40.911582 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:41.910555 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:41.910526 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:41.911033 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:41.910685 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:42.478462 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:42.478415 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:42.478680 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:42.478498 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:42.478680 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.478652 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:42.478797 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.478725 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs podName:41713217-eaa0-4eb2-b6f0-d89170d86af0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:46.478702758 +0000 UTC m=+10.966893944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs") pod "network-metrics-daemon-crw6b" (UID: "41713217-eaa0-4eb2-b6f0-d89170d86af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:42.479213 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.479189 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:42.479336 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.479257 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret podName:92704e7d-742e-4358-a920-286027d064b0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:46.479239682 +0000 UTC m=+10.967430871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret") pod "global-pull-secret-syncer-wpdjf" (UID: "92704e7d-742e-4358-a920-286027d064b0") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:42.579099 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:42.579036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:42.579306 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.579286 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:32:42.579394 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.579315 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:32:42.579394 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.579330 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nrdpg for pod openshift-network-diagnostics/network-check-target-wlfd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:42.579508 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.579402 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg podName:74174e45-7725-4b5f-91d1-43ec79896c38 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:46.579379363 +0000 UTC m=+11.067570547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrdpg" (UniqueName: "kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg") pod "network-check-target-wlfd9" (UID: "74174e45-7725-4b5f-91d1-43ec79896c38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:42.911427 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:42.911391 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:42.911909 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.911540 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:42.912059 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:42.912036 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:42.912167 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:42.912143 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:43.911143 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:43.910939 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:43.911143 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:43.911132 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:44.910798 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:44.910060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:44.910798 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:44.910204 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:44.910798 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:44.910657 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:44.910798 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:44.910751 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:45.910119 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:45.910082 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:45.910334 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:45.910248 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:46.515857 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:46.515723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:46.515857 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:46.515778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:46.516425 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.515888 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:46.516425 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.515964 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret podName:92704e7d-742e-4358-a920-286027d064b0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:54.515942235 +0000 UTC m=+19.004133431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret") pod "global-pull-secret-syncer-wpdjf" (UID: "92704e7d-742e-4358-a920-286027d064b0") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:46.516535 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.516427 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:46.516535 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.516504 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs podName:41713217-eaa0-4eb2-b6f0-d89170d86af0 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:54.516482455 +0000 UTC m=+19.004673660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs") pod "network-metrics-daemon-crw6b" (UID: "41713217-eaa0-4eb2-b6f0-d89170d86af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:46.617149 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:46.617109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:46.617348 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.617273 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:32:46.617348 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.617299 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:32:46.617348 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.617313 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nrdpg for pod openshift-network-diagnostics/network-check-target-wlfd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:46.617508 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.617377 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg podName:74174e45-7725-4b5f-91d1-43ec79896c38 nodeName:}" failed. No retries permitted until 2026-07-21 14:32:54.617360195 +0000 UTC m=+19.105551378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrdpg" (UniqueName: "kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg") pod "network-check-target-wlfd9" (UID: "74174e45-7725-4b5f-91d1-43ec79896c38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:46.912508 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:46.911781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:46.912508 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.911943 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:46.912508 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:46.912159 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:46.912508 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:46.912301 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:47.910329 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:47.910290 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:47.910808 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:47.910449 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:48.910837 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:48.910677 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:48.910837 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:48.910682 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:48.910837 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:48.910822 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:48.911446 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:48.910887 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:49.909940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:49.909902 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:49.910100 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:49.910050 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:50.910546 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:50.910500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:50.910917 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:50.910627 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:50.910917 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:50.910679 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:50.910917 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:50.910767 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:51.909931 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:51.909895 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:51.910125 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:51.910027 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:52.909681 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:52.909652 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:52.910093 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:52.909754 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:52.910093 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:52.909838 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:52.910093 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:52.909926 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:53.910259 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:53.910205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:53.910761 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:53.910369 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:54.575227 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:54.575176 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:54.575418 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:54.575342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:54.575418 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.575375 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:54.575528 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.575456 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret podName:92704e7d-742e-4358-a920-286027d064b0 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:10.575434346 +0000 UTC m=+35.063625547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret") pod "global-pull-secret-syncer-wpdjf" (UID: "92704e7d-742e-4358-a920-286027d064b0") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:32:54.575528 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.575465 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:54.575528 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.575523 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs podName:41713217-eaa0-4eb2-b6f0-d89170d86af0 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:10.575504643 +0000 UTC m=+35.063695844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs") pod "network-metrics-daemon-crw6b" (UID: "41713217-eaa0-4eb2-b6f0-d89170d86af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:32:54.675914 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:54.675868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:54.676092 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.676071 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:32:54.676145 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.676098 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:32:54.676145 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.676111 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nrdpg for pod openshift-network-diagnostics/network-check-target-wlfd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:54.676274 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.676178 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg podName:74174e45-7725-4b5f-91d1-43ec79896c38 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:10.676157925 +0000 UTC m=+35.164349108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrdpg" (UniqueName: "kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg") pod "network-check-target-wlfd9" (UID: "74174e45-7725-4b5f-91d1-43ec79896c38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:32:54.910617 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:54.910529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:54.911042 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.910678 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:54.911042 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:54.910722 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:54.911042 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:54.910849 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:55.910115 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:55.910074 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:55.910294 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:55.910234 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:56.911316 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:56.911276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:56.911786 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:56.911298 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:56.911786 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:56.911413 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:56.911786 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:56.911483 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:57.909960 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:57.909933 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:57.910145 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:57.910045 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:32:58.910179 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:58.910144 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:32:58.910658 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:58.910276 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:32:58.910658 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:58.910325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:32:58.910658 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:58.910417 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:32:59.910735 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:32:59.910601 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:32:59.911008 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:32:59.910838 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:00.031265 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.031234 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sdfsq" event={"ID":"db2d40c8-02bb-461d-b2da-77af6c63dc7a","Type":"ContainerStarted","Data":"548e1dd66f0bbbf0a612c62d964fb4f477e1407ab5d4db9bb37d1e52a2b8ac2b"} Jul 21 14:33:00.033390 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.033366 2559 generic.go:356] "Generic (PLEG): container finished" podID="34fbe162-4fcd-4382-9b5d-eb2873fe641c" containerID="6b78354baafbdad056f91aaf5fb7b760e99499a15456a8a00b7560dc1b251950" exitCode=0 Jul 21 14:33:00.033500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.033433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerDied","Data":"6b78354baafbdad056f91aaf5fb7b760e99499a15456a8a00b7560dc1b251950"} Jul 21 14:33:00.034930 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.034814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" event={"ID":"dfd1b571-fb06-4715-8e64-96e42a20a05d","Type":"ContainerStarted","Data":"fc99dcb8770f7de662f7f66b7493079b80ba4ae3a064c32d595f523efd0e2940"} Jul 21 14:33:00.036276 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.036254 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wkxm" event={"ID":"2e9595f3-e87e-46b8-993b-9fb4981d50d8","Type":"ContainerStarted","Data":"9372e2ee76d38475d467e92c07f913c9b7d7048c28df88a0dc1b30f945758245"} Jul 21 14:33:00.037585 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.037565 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" event={"ID":"abb41b77-8b49-4565-817b-76b89de2ea8a","Type":"ContainerStarted","Data":"3de8be8d3ac3821cecfd9fce882b032e802a1deebea11ca49366a30ea498ec7e"} Jul 21 14:33:00.039311 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.039294 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:33:00.039621 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.039604 2559 generic.go:356] "Generic (PLEG): container finished" podID="e5287c9f-3012-4c0b-bed8-3cd0af5c365b" containerID="c4a191511359134684743502ac95bee84be7f1426d6b0286a98e4cddba3f0add" exitCode=1 Jul 21 14:33:00.039704 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.039669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerDied","Data":"c4a191511359134684743502ac95bee84be7f1426d6b0286a98e4cddba3f0add"} Jul 21 14:33:00.039755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.039739 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"2f7243c14db04d485a792cbe559481eebe616b99d0e8aab4c007d18cef4e16f1"} Jul 21 14:33:00.040981 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.040962 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" event={"ID":"1d033722713bf948e6f973d6228bc956","Type":"ContainerStarted","Data":"8810921e5f0c32cab94c3a874727f9021e83f476ac9a06f6b6f1643df755a04d"} Jul 21 14:33:00.042402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.042373 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lc8kv" event={"ID":"420a38b5-47bb-4326-927f-cc0a80bde4c4","Type":"ContainerStarted","Data":"9c88692c9b36acc73de3162289fa93e13eca14b1c18be13b288667cb98ae7448"} Jul 21 14:33:00.043846 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.043825 2559 generic.go:356] "Generic (PLEG): container finished" podID="27e815f37a7bba8db1a0a898d0836892" containerID="ec7fced1348038db2ba304330ab65df39aed639b9d16116eeb4c7571b4e5b733" exitCode=0 Jul 21 14:33:00.043948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.043886 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" event={"ID":"27e815f37a7bba8db1a0a898d0836892","Type":"ContainerDied","Data":"ec7fced1348038db2ba304330ab65df39aed639b9d16116eeb4c7571b4e5b733"} Jul 21 14:33:00.045321 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.045300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9mjkx" event={"ID":"96be4230-5b49-4819-b027-1c57fc60964d","Type":"ContainerStarted","Data":"e9abf70676f33894c1fa29dccb095f484e840f35f986feecb2e0f6056552f293"} Jul 21 14:33:00.051016 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.050974 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sdfsq" podStartSLOduration=2.888378167 podStartE2EDuration="23.050962947s" totalImagesPullingTime="20.16258478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:00.050184986 +0000 UTC m=+24.538376186" watchObservedRunningTime="2026-07-21 14:33:00.050962947 +0000 UTC m=+24.539154151" Jul 21 14:33:00.065159 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.065114 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9mjkx" podStartSLOduration=7.398994037 podStartE2EDuration="23.065096436s" totalImagesPullingTime="15.666102399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:00.064577836 +0000 UTC m=+24.552769037" watchObservedRunningTime="2026-07-21 14:33:00.065096436 +0000 UTC m=+24.553287644" Jul 21 14:33:00.095042 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.094994 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lc8kv" podStartSLOduration=2.99288288 podStartE2EDuration="23.094974473s" totalImagesPullingTime="20.102091593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:00.094935499 +0000 UTC m=+24.583126729" watchObservedRunningTime="2026-07-21 14:33:00.094974473 +0000 UTC m=+24.583165657" Jul 21 14:33:00.095374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.095336 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8wkxm" podStartSLOduration=3.012798393 podStartE2EDuration="23.095323841s" totalImagesPullingTime="20.082525448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:00.079873392 +0000 UTC m=+24.568064615" watchObservedRunningTime="2026-07-21 14:33:00.095323841 +0000 UTC m=+24.583515045" Jul 21 14:33:00.122884 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.122836 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-112.ec2.internal" podStartSLOduration=22.122819214 podStartE2EDuration="22.122819214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:00.122091256 +0000 UTC m=+24.610282479" watchObservedRunningTime="2026-07-21 14:33:00.122819214 +0000 UTC m=+24.611010419" Jul 21 14:33:00.137836 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.137796 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q6c9v" podStartSLOduration=2.977942124 podStartE2EDuration="23.137780698s" totalImagesPullingTime="20.159838574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:00.137193927 +0000 UTC m=+24.625385130" watchObservedRunningTime="2026-07-21 14:33:00.137780698 +0000 UTC m=+24.625971952" Jul 21 14:33:00.614490 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.614413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:00.696474 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.696437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:00.757929 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.757897 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:00.909995 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.909907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:00.910154 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:00.909909 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:00.910154 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:00.910053 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:00.910154 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:00.910113 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:01.039375 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.039337 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:01.051037 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.051010 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:33:01.051500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.051470 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"ed98ee4a55d12a99f6cd068b4865572aa94bdd06948972293bf2918a652edc5f"} Jul 21 14:33:01.051615 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.051557 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"998ec890c903d7120cf9d139d118745226bcd1389f540431791f500dc77be216"} Jul 21 14:33:01.051615 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.051608 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"37cdc4850d04255f4620f0a45c390bb95b7b87e0b8c7dbadc7f91d90a1a6ccba"} Jul 21 14:33:01.051736 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.051654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"12d167910f4e2f30281b3bc49666b7afc72da1d20e13ad248885f314f4253c71"} Jul 21 14:33:01.053690 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.053663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" event={"ID":"27e815f37a7bba8db1a0a898d0836892","Type":"ContainerStarted","Data":"b39462bd204051e655d699d241dcc35aaf100986eaa6ea5bb1b7a0a1ba4e3938"} Jul 21 14:33:01.055378 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.055338 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-snbnj" event={"ID":"070e8201-fcbc-45cc-b266-1b3625351d2d","Type":"ContainerStarted","Data":"e89a034a9b27947fdecd80850326a25b46ce7322fab1b8d854f803087b667282"} Jul 21 14:33:01.068574 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.068523 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-112.ec2.internal" podStartSLOduration=23.068506386 podStartE2EDuration="23.068506386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:01.068381857 +0000 UTC m=+25.556573072" watchObservedRunningTime="2026-07-21 14:33:01.068506386 +0000 UTC m=+25.556697593" Jul 21 14:33:01.082566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.082524 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-snbnj" podStartSLOduration=3.996391557 podStartE2EDuration="24.082510814s" totalImagesPullingTime="20.086119257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:01.082155685 +0000 UTC m=+25.570346889" watchObservedRunningTime="2026-07-21 14:33:01.082510814 +0000 UTC m=+25.570702018" Jul 21 14:33:01.140105 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.140070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:01.450240 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.450176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:01.555041 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.555007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:01.891271 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.891237 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:01.910512 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:01.910474 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:01.910682 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:01.910601 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:02.357232 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:02.357177 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:02.458100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:02.458067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:02.849853 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:02.849826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:02.912822 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:02.912798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:02.912822 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:02.912820 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:02.913001 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:02.912913 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:02.913001 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:02.912983 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:02.920278 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:02.920255 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 14:33:02.950351 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:02.950311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:03.061182 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.061147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" event={"ID":"abb41b77-8b49-4565-817b-76b89de2ea8a","Type":"ContainerStarted","Data":"5b9e722ee14a2f1135f252d1b4e84da2f2c3ed276d50780db82bbb693d5b3aee"} Jul 21 14:33:03.064036 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.064008 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:33:03.064375 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.064355 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"2b77c35ba38079e781568d926b8b2d13742264d18eaccc7328d595f3f47301cd"} Jul 21 14:33:03.666577 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.666542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:03.765886 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.765850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:03.810313 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.810173 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T14:33:02.920272949Z","UUID":"49132616-20f7-4498-a0e9-579a7f9a5d37","Handler":null,"Name":"","Endpoint":""} Jul 21 14:33:03.811859 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.811832 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 14:33:03.811999 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.811881 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 14:33:03.909721 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:03.909685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:03.909917 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:03.909832 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:04.115888 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:04.115855 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:33:04.116542 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:04.116519 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:33:04.910199 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:04.910167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:04.910622 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:04.910278 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:04.910622 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:04.910342 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:04.910622 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:04.910447 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:05.017611 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.017428 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.073027 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.073007 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:33:05.073608 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.073438 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"dfb285fccf20160f98c44d3c9fb891eab07e8069814a1776bb8b3e132856ef15"} Jul 21 14:33:05.073809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.073729 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:33:05.074092 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.074068 2559 scope.go:117] "RemoveContainer" containerID="c4a191511359134684743502ac95bee84be7f1426d6b0286a98e4cddba3f0add" Jul 21 14:33:05.074489 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.074283 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lc8kv" Jul 21 14:33:05.105593 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.105572 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.198028 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.197747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.297826 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.297790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.316143 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.316119 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:33:05.397732 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.397701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.455982 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.455898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.518980 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.518939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.714953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.714874 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:05.910403 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.910372 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:05.911081 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:05.910474 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:05.992646 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:05.992623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:06.076819 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.076790 2559 generic.go:356] "Generic (PLEG): container finished" podID="34fbe162-4fcd-4382-9b5d-eb2873fe641c" containerID="51b7a1c5aa4415f5c24e98c1d3c7af8e3724367ed1c7df8f71d6831758b4e37a" exitCode=0 Jul 21 14:33:06.076926 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.076844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerDied","Data":"51b7a1c5aa4415f5c24e98c1d3c7af8e3724367ed1c7df8f71d6831758b4e37a"} Jul 21 14:33:06.080623 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.080607 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:33:06.080906 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.080887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" event={"ID":"e5287c9f-3012-4c0b-bed8-3cd0af5c365b","Type":"ContainerStarted","Data":"485fb60b0e819939704a7c69b871e305407f688e2111e56e6bf00e63b7b278f5"} Jul 21 14:33:06.081096 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.081076 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:33:06.081176 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.081147 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:33:06.095298 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.095278 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:33:06.095579 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.095567 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:33:06.124773 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.124733 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" podStartSLOduration=8.6365278 podStartE2EDuration="29.1247201s" totalImagesPullingTime="20.4881923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:06.123928478 +0000 UTC m=+30.612119682" watchObservedRunningTime="2026-07-21 14:33:06.1247201 +0000 UTC m=+30.612911301" Jul 21 14:33:06.525431 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.525402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:06.910420 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.910389 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:06.910816 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.910465 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:06.910816 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:06.910471 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:06.910816 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:06.910553 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:06.925204 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:06.925172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:07.085637 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:07.085431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" event={"ID":"abb41b77-8b49-4565-817b-76b89de2ea8a","Type":"ContainerStarted","Data":"0959774bc8e92abfdd1bc783073ec253ebc795d6a4fd03e3e43f06476a591a66"} Jul 21 14:33:07.108100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:07.108041 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5gm8" podStartSLOduration=4.047078912 podStartE2EDuration="30.108025054s" totalImagesPullingTime="26.060946142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:07.107719456 +0000 UTC m=+31.595910660" watchObservedRunningTime="2026-07-21 14:33:07.108025054 +0000 UTC m=+31.596216258" Jul 21 14:33:07.314619 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:07.314593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:07.515019 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:07.514994 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:07.910009 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:07.909976 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:07.910177 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:07.910081 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:07.913653 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:07.913631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:08.089605 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:08.089575 2559 generic.go:356] "Generic (PLEG): container finished" podID="34fbe162-4fcd-4382-9b5d-eb2873fe641c" containerID="0d235eca1d5d1b982b7c2c977a59cc1806aadbd1b6e68406fe431a5a4d99ec9d" exitCode=0 Jul 21 14:33:08.089770 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:08.089659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerDied","Data":"0d235eca1d5d1b982b7c2c977a59cc1806aadbd1b6e68406fe431a5a4d99ec9d"} Jul 21 14:33:08.537914 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:08.537876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:08.715132 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:08.715098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:08.823596 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:08.823516 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:08.910802 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:08.910766 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:08.910802 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:08.910794 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:08.910968 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:08.910919 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:08.911043 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:08.911011 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:09.130165 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:09.130102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:09.312766 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:09.312732 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:09.712752 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:09.712674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:09.767888 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:09.767851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:09.910361 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:09.910325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:09.910605 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:09.910432 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:09.914790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:09.914762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:10.095729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.095699 2559 generic.go:356] "Generic (PLEG): container finished" podID="34fbe162-4fcd-4382-9b5d-eb2873fe641c" containerID="fabec5db70f2721457fb3c6dc879011b92e020af4a37016cf729ef9bd4b16b40" exitCode=0 Jul 21 14:33:10.095899 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.095755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerDied","Data":"fabec5db70f2721457fb3c6dc879011b92e020af4a37016cf729ef9bd4b16b40"} Jul 21 14:33:10.518454 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.518374 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:10.590331 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.590305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:10.590476 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.590341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:10.590476 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.590434 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:33:10.590476 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.590445 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:33:10.590476 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.590478 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs podName:41713217-eaa0-4eb2-b6f0-d89170d86af0 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:42.590464646 +0000 UTC m=+67.078655829 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs") pod "network-metrics-daemon-crw6b" (UID: "41713217-eaa0-4eb2-b6f0-d89170d86af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:33:10.590632 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.590502 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret podName:92704e7d-742e-4358-a920-286027d064b0 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:42.590483439 +0000 UTC m=+67.078674642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret") pod "global-pull-secret-syncer-wpdjf" (UID: "92704e7d-742e-4358-a920-286027d064b0") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:33:10.690628 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.690597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:10.690746 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.690712 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:33:10.690746 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.690728 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:33:10.690746 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.690736 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nrdpg for pod openshift-network-diagnostics/network-check-target-wlfd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:33:10.690839 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.690778 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg podName:74174e45-7725-4b5f-91d1-43ec79896c38 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:42.6907665 +0000 UTC m=+67.178957695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrdpg" (UniqueName: "kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg") pod "network-check-target-wlfd9" (UID: "74174e45-7725-4b5f-91d1-43ec79896c38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:33:10.910271 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.910240 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:10.910271 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.910274 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:10.910453 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.910350 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:10.910487 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:10.910460 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:10.918076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:10.918053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:11.315274 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:11.315241 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:11.415818 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:11.415783 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:11.515907 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:11.515872 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:11.910213 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:11.910184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:11.910602 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:11.910317 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:11.913560 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:11.913531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:12.715378 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:12.715294 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:12.910173 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:12.910123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:12.910356 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:12.910295 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:12.910356 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:12.910331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:12.910686 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:12.910454 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:12.964355 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:12.964322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:13.315720 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:13.315681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:13.525150 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:13.525120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:13.909838 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:13.909790 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:13.910021 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:13.909941 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:13.918796 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:13.918761 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:14.395006 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:14.394972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:14.716258 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:14.716162 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:14.910510 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:14.910479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:14.910510 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:14.910498 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:14.910744 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:14.910601 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:14.910805 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:14.910734 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:15.187800 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:15.187766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:15.314527 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:15.314495 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:15.520995 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:15.520954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:15.717456 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:15.717357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:15.909673 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:15.909636 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:15.909851 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:15.909746 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:16.111966 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:16.111935 2559 generic.go:356] "Generic (PLEG): container finished" podID="34fbe162-4fcd-4382-9b5d-eb2873fe641c" containerID="7a388ea7396996070222b153713c49b5521f5ab050bf979d215f8063b62886cd" exitCode=0 Jul 21 14:33:16.112148 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:16.111998 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerDied","Data":"7a388ea7396996070222b153713c49b5521f5ab050bf979d215f8063b62886cd"} Jul 21 14:33:16.910427 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:16.910392 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:16.910807 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:16.910463 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:16.910807 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:16.910482 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:16.910807 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:16.910543 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:17.116334 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:17.116299 2559 generic.go:356] "Generic (PLEG): container finished" podID="34fbe162-4fcd-4382-9b5d-eb2873fe641c" containerID="c79d4abb9478a98f177b193b49468b862454aa62caf1ad48fb134132dc4785ad" exitCode=0 Jul 21 14:33:17.116485 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:17.116358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerDied","Data":"c79d4abb9478a98f177b193b49468b862454aa62caf1ad48fb134132dc4785ad"} Jul 21 14:33:17.910664 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:17.910632 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:17.911113 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:17.910731 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:18.120971 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:18.120938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpksb" event={"ID":"34fbe162-4fcd-4382-9b5d-eb2873fe641c","Type":"ContainerStarted","Data":"b38c5e38c47accd49f169a04c7069f3d70828ee4672a377b88bd5f74db6a0fc5"} Jul 21 14:33:18.143637 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:18.143591 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hpksb" podStartSLOduration=8.124958261 podStartE2EDuration="41.143576436s" totalImagesPullingTime="33.018618175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:18.142028858 +0000 UTC m=+42.630220081" watchObservedRunningTime="2026-07-21 14:33:18.143576436 +0000 UTC m=+42.631767639" Jul 21 14:33:18.909940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:18.909900 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:18.910141 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:18.909900 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:18.910141 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:18.910014 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:18.910141 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:18.910077 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:19.910298 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:19.910261 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:19.910679 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:19.910373 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:20.910154 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:20.910120 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:20.910368 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:20.910122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:20.910368 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:20.910268 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:20.910368 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:20.910303 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:21.877080 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:21.877047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:21.910398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:21.910363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:21.910818 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:21.910501 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:22.010490 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.010458 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:22.229256 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.229146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:22.419758 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.419721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:22.506428 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.506354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:22.583333 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.583305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:22.910484 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.910447 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:22.910937 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.910455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:22.910937 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:22.910568 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:22.910937 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:22.910628 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:22.922161 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:22.922119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:23.209230 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.208951 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlfd9"] Jul 21 14:33:23.209378 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.209256 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:23.209378 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:23.209347 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:23.211841 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.211802 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wpdjf"] Jul 21 14:33:23.211965 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.211904 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:23.212029 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:23.212006 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:23.212498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.212476 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crw6b"] Jul 21 14:33:23.212606 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.212592 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:23.212721 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:23.212700 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:23.428208 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.428166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:23.528599 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.528513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:23.748371 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:23.748338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:24.616540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:24.616500 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:24.910045 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:24.909967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:24.910188 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:24.910063 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:24.910188 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:24.909967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:24.910188 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:24.910168 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:24.910316 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:24.910206 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:24.910316 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:24.910267 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:24.951322 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:24.951295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:25.018144 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:25.018107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:25.086718 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:25.086687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:26.456410 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:26.456378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:26.522684 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:26.522614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:26.911346 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:26.911316 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:26.911504 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:26.911387 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:26.911504 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:26.911405 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crw6b" podUID="41713217-eaa0-4eb2-b6f0-d89170d86af0" Jul 21 14:33:26.911504 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:26.911486 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpdjf" podUID="92704e7d-742e-4358-a920-286027d064b0" Jul 21 14:33:26.911668 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:26.911514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:26.911668 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:26.911559 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlfd9" podUID="74174e45-7725-4b5f-91d1-43ec79896c38" Jul 21 14:33:27.243663 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.243585 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-112.ec2.internal" event="NodeReady" Jul 21 14:33:27.243784 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.243700 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 14:33:27.303589 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.303555 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tfp45"] Jul 21 14:33:27.322402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.322367 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r9s6p"] Jul 21 14:33:27.322550 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.322511 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.325522 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.325495 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 14:33:27.325791 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.325765 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 14:33:27.326345 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.326321 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2qjq\"" Jul 21 14:33:27.346969 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.346946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tfp45"] Jul 21 14:33:27.347082 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.346986 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r9s6p"] Jul 21 14:33:27.347082 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.347074 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.351150 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.351130 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 14:33:27.351286 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.351242 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wkr9m\"" Jul 21 14:33:27.351286 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.351281 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 14:33:27.351525 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.351505 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 14:33:27.351668 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.351652 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 14:33:27.409073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.409039 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7rjrn"] Jul 21 14:33:27.422755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.422724 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rjrn"] Jul 21 14:33:27.422755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.422747 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.425391 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.425365 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 14:33:27.425562 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.425542 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 14:33:27.425654 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.425622 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 14:33:27.425737 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.425721 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-prcnc\"" Jul 21 14:33:27.497486 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.497399 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:27.523515 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523488 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-tmp-dir\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.523635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4352a4a4-059c-433d-ba11-81053151750c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.523635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523541 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-metrics-tls\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.523635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523584 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130ab0a9-4626-4c0a-adcf-e715b2cf52ea-cert\") pod \"ingress-canary-7rjrn\" (UID: \"130ab0a9-4626-4c0a-adcf-e715b2cf52ea\") " pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.523635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6ft\" (UniqueName: \"kubernetes.io/projected/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-kube-api-access-jl6ft\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.523755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4352a4a4-059c-433d-ba11-81053151750c-crio-socket\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.523755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-config-volume\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.523755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmb7h\" (UniqueName: \"kubernetes.io/projected/130ab0a9-4626-4c0a-adcf-e715b2cf52ea-kube-api-access-mmb7h\") pod \"ingress-canary-7rjrn\" (UID: \"130ab0a9-4626-4c0a-adcf-e715b2cf52ea\") " pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.523844 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523756 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4352a4a4-059c-433d-ba11-81053151750c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.523844 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktd4\" (UniqueName: \"kubernetes.io/projected/4352a4a4-059c-433d-ba11-81053151750c-kube-api-access-sktd4\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.523844 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.523819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4352a4a4-059c-433d-ba11-81053151750c-data-volume\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.597442 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.597410 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:27.624428 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624402 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6ft\" (UniqueName: \"kubernetes.io/projected/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-kube-api-access-jl6ft\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.624578 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4352a4a4-059c-433d-ba11-81053151750c-crio-socket\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.624578 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624505 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-config-volume\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.624578 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmb7h\" (UniqueName: \"kubernetes.io/projected/130ab0a9-4626-4c0a-adcf-e715b2cf52ea-kube-api-access-mmb7h\") pod \"ingress-canary-7rjrn\" (UID: \"130ab0a9-4626-4c0a-adcf-e715b2cf52ea\") " pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.624711 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4352a4a4-059c-433d-ba11-81053151750c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.624711 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4352a4a4-059c-433d-ba11-81053151750c-crio-socket\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.624814 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktd4\" (UniqueName: \"kubernetes.io/projected/4352a4a4-059c-433d-ba11-81053151750c-kube-api-access-sktd4\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.624814 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4352a4a4-059c-433d-ba11-81053151750c-data-volume\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.624919 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-tmp-dir\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.624973 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4352a4a4-059c-433d-ba11-81053151750c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.624973 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-metrics-tls\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.625078 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.624974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130ab0a9-4626-4c0a-adcf-e715b2cf52ea-cert\") pod \"ingress-canary-7rjrn\" (UID: \"130ab0a9-4626-4c0a-adcf-e715b2cf52ea\") " pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.625078 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.625043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-config-volume\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.625078 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.625054 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4352a4a4-059c-433d-ba11-81053151750c-data-volume\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.625323 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.625304 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-tmp-dir\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.625365 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.625334 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4352a4a4-059c-433d-ba11-81053151750c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.628934 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.628906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4352a4a4-059c-433d-ba11-81053151750c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.629036 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.628935 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130ab0a9-4626-4c0a-adcf-e715b2cf52ea-cert\") pod \"ingress-canary-7rjrn\" (UID: \"130ab0a9-4626-4c0a-adcf-e715b2cf52ea\") " pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.629036 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.628955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-metrics-tls\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.637917 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.637876 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktd4\" (UniqueName: \"kubernetes.io/projected/4352a4a4-059c-433d-ba11-81053151750c-kube-api-access-sktd4\") pod \"insights-runtime-extractor-r9s6p\" (UID: \"4352a4a4-059c-433d-ba11-81053151750c\") " pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.638041 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.637975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6ft\" (UniqueName: \"kubernetes.io/projected/bd5ce839-4b3a-4927-a991-1aa5cfad8af1-kube-api-access-jl6ft\") pod \"dns-default-tfp45\" (UID: \"bd5ce839-4b3a-4927-a991-1aa5cfad8af1\") " pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:27.639471 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.639444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmb7h\" (UniqueName: \"kubernetes.io/projected/130ab0a9-4626-4c0a-adcf-e715b2cf52ea-kube-api-access-mmb7h\") pod \"ingress-canary-7rjrn\" (UID: \"130ab0a9-4626-4c0a-adcf-e715b2cf52ea\") " pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.655321 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.655296 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r9s6p" Jul 21 14:33:27.731824 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.731787 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rjrn" Jul 21 14:33:27.852046 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.850683 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r9s6p"] Jul 21 14:33:27.899095 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.898533 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rjrn"] Jul 21 14:33:27.901134 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:27.901109 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130ab0a9_4626_4c0a_adcf_e715b2cf52ea.slice/crio-8b3e547d2c1039e4e55d422a8abb1593a4eafb4b5c255bcac5ed41162400bbe7 WatchSource:0}: Error finding container 8b3e547d2c1039e4e55d422a8abb1593a4eafb4b5c255bcac5ed41162400bbe7: Status 404 returned error can't find the container with id 8b3e547d2c1039e4e55d422a8abb1593a4eafb4b5c255bcac5ed41162400bbe7 Jul 21 14:33:27.932892 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:27.932857 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:28.057659 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.057630 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tfp45"] Jul 21 14:33:28.060706 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:28.060678 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5ce839_4b3a_4927_a991_1aa5cfad8af1.slice/crio-9deb435f537bdce54402346c68dfc8f7c987c4ee3b6f97a3d20f85b69703390d WatchSource:0}: Error finding container 9deb435f537bdce54402346c68dfc8f7c987c4ee3b6f97a3d20f85b69703390d: Status 404 returned error can't find the container with id 9deb435f537bdce54402346c68dfc8f7c987c4ee3b6f97a3d20f85b69703390d Jul 21 14:33:28.146152 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.146123 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfp45" event={"ID":"bd5ce839-4b3a-4927-a991-1aa5cfad8af1","Type":"ContainerStarted","Data":"9deb435f537bdce54402346c68dfc8f7c987c4ee3b6f97a3d20f85b69703390d"} Jul 21 14:33:28.147156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.147133 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rjrn" event={"ID":"130ab0a9-4626-4c0a-adcf-e715b2cf52ea","Type":"ContainerStarted","Data":"8b3e547d2c1039e4e55d422a8abb1593a4eafb4b5c255bcac5ed41162400bbe7"} Jul 21 14:33:28.148166 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.148144 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r9s6p" event={"ID":"4352a4a4-059c-433d-ba11-81053151750c","Type":"ContainerStarted","Data":"7a78f277648bdb75bd55637f54c80dc80f802fa1b88e5914fa3598e36e48274e"} Jul 21 14:33:28.910374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.910339 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:28.911505 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.911161 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:28.911756 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.911734 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:28.915851 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.915125 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 14:33:28.915851 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.915364 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c99qd\"" Jul 21 14:33:28.915851 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.915559 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 14:33:28.915851 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.915726 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cprh8\"" Jul 21 14:33:28.916103 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.915888 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 14:33:28.916158 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:28.916134 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 14:33:30.704654 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.704607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:30.779465 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.779430 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-846d44694f-5sjdx"] Jul 21 14:33:30.781544 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.781517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.784175 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.784155 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 14:33:30.784300 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.784195 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 14:33:30.785329 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.785306 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fknrw\"" Jul 21 14:33:30.785547 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.785529 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 14:33:30.785622 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.785546 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 14:33:30.785622 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.785560 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 14:33:30.785728 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.785660 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 14:33:30.785907 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.785885 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 14:33:30.792675 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.792654 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846d44694f-5sjdx"] Jul 21 14:33:30.806411 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.806390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:30.853069 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.853031 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-config\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.853237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.853117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-service-ca\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.853237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.853204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-oauth-config\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.853383 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.853271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-serving-cert\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.853383 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.853298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-oauth-serving-cert\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.853383 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.853352 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkl6\" (UniqueName: \"kubernetes.io/projected/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-kube-api-access-tjkl6\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.914421 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.914395 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh"] Jul 21 14:33:30.915475 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.915448 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:30.916493 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.916475 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:30.918945 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.918926 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 14:33:30.919050 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.919007 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mvqjl\"" Jul 21 14:33:30.926023 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.925986 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh"] Jul 21 14:33:30.957469 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.954491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-oauth-config\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.957647 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.957622 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-serving-cert\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.957703 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.957675 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-oauth-serving-cert\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.957868 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.957841 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkl6\" (UniqueName: \"kubernetes.io/projected/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-kube-api-access-tjkl6\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.958380 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.958359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94aafa15-2b91-4c73-b51b-e622db2bf7c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-hx8jh\" (UID: \"94aafa15-2b91-4c73-b51b-e622db2bf7c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:30.958546 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.958417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-config\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.958546 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.958448 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-service-ca\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.959043 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.959017 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-oauth-serving-cert\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.959197 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.959176 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-service-ca\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.959313 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.959272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-config\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.959878 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.959851 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-oauth-config\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.961092 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.961073 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-serving-cert\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:30.968442 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:30.968423 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkl6\" (UniqueName: \"kubernetes.io/projected/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-kube-api-access-tjkl6\") pod \"console-846d44694f-5sjdx\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:31.059961 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:31.059786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94aafa15-2b91-4c73-b51b-e622db2bf7c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-hx8jh\" (UID: \"94aafa15-2b91-4c73-b51b-e622db2bf7c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:31.059961 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:31.059947 2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 21 14:33:31.060169 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:31.060039 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94aafa15-2b91-4c73-b51b-e622db2bf7c3-tls-certificates podName:94aafa15-2b91-4c73-b51b-e622db2bf7c3 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:31.560017272 +0000 UTC m=+56.048208470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/94aafa15-2b91-4c73-b51b-e622db2bf7c3-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-hx8jh" (UID: "94aafa15-2b91-4c73-b51b-e622db2bf7c3") : secret "prometheus-operator-admission-webhook-tls" not found Jul 21 14:33:31.093320 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:31.093278 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:31.159135 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:31.159104 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfp45" event={"ID":"bd5ce839-4b3a-4927-a991-1aa5cfad8af1","Type":"ContainerStarted","Data":"00159b2e4616cfa7be0eec9d6d05ffdc0d37936092803525552fa862e6a90207"} Jul 21 14:33:31.411459 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:31.411424 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:31.563142 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:31.563112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94aafa15-2b91-4c73-b51b-e622db2bf7c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-hx8jh\" (UID: \"94aafa15-2b91-4c73-b51b-e622db2bf7c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:31.566160 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:31.566135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94aafa15-2b91-4c73-b51b-e622db2bf7c3-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-hx8jh\" (UID: \"94aafa15-2b91-4c73-b51b-e622db2bf7c3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:31.826447 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:31.826418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:32.710203 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.710174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:32.755571 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.755531 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6695478675-mckkj"] Jul 21 14:33:32.772295 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.772261 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6695478675-mckkj"] Jul 21 14:33:32.772487 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.772375 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.780545 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.780520 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 14:33:32.807567 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.807527 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:32.873480 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.873385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-config\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.873480 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.873455 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-service-ca\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.873934 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.873491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gn92\" (UniqueName: \"kubernetes.io/projected/9cb85d09-9d20-4aa3-8817-2f795fd49484-kube-api-access-2gn92\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.873934 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.873517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-trusted-ca-bundle\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.873934 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.873537 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-oauth-serving-cert\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.873934 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.873651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-oauth-config\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.873934 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.873695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-serving-cert\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.874361 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.874339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:32.974555 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.974530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-service-ca\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.974626 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.974564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gn92\" (UniqueName: \"kubernetes.io/projected/9cb85d09-9d20-4aa3-8817-2f795fd49484-kube-api-access-2gn92\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.974626 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.974597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-trusted-ca-bundle\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.974795 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.974769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-oauth-serving-cert\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.974933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.974821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-oauth-config\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.974933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.974869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-serving-cert\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.974933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.974921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-config\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.975555 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.975533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-service-ca\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.975632 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.975533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-oauth-serving-cert\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.975632 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.975602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-config\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.975632 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.975612 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-trusted-ca-bundle\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.977764 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.977740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-serving-cert\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.977867 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.977778 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-oauth-config\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:32.984688 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:32.984659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gn92\" (UniqueName: \"kubernetes.io/projected/9cb85d09-9d20-4aa3-8817-2f795fd49484-kube-api-access-2gn92\") pod \"console-6695478675-mckkj\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:33.085981 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:33.085928 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:33.721001 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:33.720896 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:33.814893 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:33.814645 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh"] Jul 21 14:33:33.817089 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:33.817061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:33.817574 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:33.817544 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94aafa15_2b91_4c73_b51b_e622db2bf7c3.slice/crio-6a53088a604d8c9193bb1efb7f9bae08a8f41e297a772799a55a291fa9f1e46c WatchSource:0}: Error finding container 6a53088a604d8c9193bb1efb7f9bae08a8f41e297a772799a55a291fa9f1e46c: Status 404 returned error can't find the container with id 6a53088a604d8c9193bb1efb7f9bae08a8f41e297a772799a55a291fa9f1e46c Jul 21 14:33:33.825675 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:33.825613 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846d44694f-5sjdx"] Jul 21 14:33:33.829381 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:33.829354 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0631a7_8761_4ae0_9bc6_4b7813af7f71.slice/crio-fce986bcc4eb68cea024c4707376e7656fcd343e0da43c08fb3848878051c2e0 WatchSource:0}: Error finding container fce986bcc4eb68cea024c4707376e7656fcd343e0da43c08fb3848878051c2e0: Status 404 returned error can't find the container with id fce986bcc4eb68cea024c4707376e7656fcd343e0da43c08fb3848878051c2e0 Jul 21 14:33:33.847728 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:33.847656 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6695478675-mckkj"] Jul 21 14:33:33.851781 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:33.851742 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb85d09_9d20_4aa3_8817_2f795fd49484.slice/crio-0106744122ba60843420ff6b72d8103f5e707897de32e78b7bb10200b867813e WatchSource:0}: Error finding container 0106744122ba60843420ff6b72d8103f5e707897de32e78b7bb10200b867813e: Status 404 returned error can't find the container with id 0106744122ba60843420ff6b72d8103f5e707897de32e78b7bb10200b867813e Jul 21 14:33:34.172782 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.172745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfp45" event={"ID":"bd5ce839-4b3a-4927-a991-1aa5cfad8af1","Type":"ContainerStarted","Data":"86171ecf2dc86f2fe7c7d160ac1bc2537550f7e5e9409eecee5d2569d610fa18"} Jul 21 14:33:34.173373 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.172858 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:34.174031 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.174011 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6695478675-mckkj" event={"ID":"9cb85d09-9d20-4aa3-8817-2f795fd49484","Type":"ContainerStarted","Data":"0106744122ba60843420ff6b72d8103f5e707897de32e78b7bb10200b867813e"} Jul 21 14:33:34.175237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.175202 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r9s6p" event={"ID":"4352a4a4-059c-433d-ba11-81053151750c","Type":"ContainerStarted","Data":"da39bee8b3348265246a6d9fd3681eb98771909fb82fe76696e8a0dc8f98a55e"} Jul 21 14:33:34.176240 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.176204 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846d44694f-5sjdx" event={"ID":"4f0631a7-8761-4ae0-9bc6-4b7813af7f71","Type":"ContainerStarted","Data":"fce986bcc4eb68cea024c4707376e7656fcd343e0da43c08fb3848878051c2e0"} Jul 21 14:33:34.177190 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.177169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" event={"ID":"94aafa15-2b91-4c73-b51b-e622db2bf7c3","Type":"ContainerStarted","Data":"6a53088a604d8c9193bb1efb7f9bae08a8f41e297a772799a55a291fa9f1e46c"} Jul 21 14:33:34.178299 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.178281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rjrn" event={"ID":"130ab0a9-4626-4c0a-adcf-e715b2cf52ea","Type":"ContainerStarted","Data":"fc2dea8465b238d44ef284646e49412415d72548a69d72e2f0834a55e4c39dac"} Jul 21 14:33:34.189486 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.189441 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tfp45" podStartSLOduration=5.127977409 podStartE2EDuration="7.189426243s" totalImagesPullingTime="2.061448834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:34.189393475 +0000 UTC m=+58.677584682" watchObservedRunningTime="2026-07-21 14:33:34.189426243 +0000 UTC m=+58.677617449" Jul 21 14:33:34.203774 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:34.203734 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7rjrn" podStartSLOduration=1.464410226 podStartE2EDuration="7.203722544s" totalImagesPullingTime="5.739312318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:34.203233676 +0000 UTC m=+58.691424898" watchObservedRunningTime="2026-07-21 14:33:34.203722544 +0000 UTC m=+58.691913726" Jul 21 14:33:36.151980 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.151929 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:36.189876 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.189819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" event={"ID":"94aafa15-2b91-4c73-b51b-e622db2bf7c3","Type":"ContainerStarted","Data":"0844ab929e2143903d7fb06aa7f25f5f80eb7b393971729e7c262a4251b7f3bc"} Jul 21 14:33:36.190137 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.190114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:36.196514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.196455 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" Jul 21 14:33:36.207800 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.207064 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hx8jh" podStartSLOduration=4.331262475 podStartE2EDuration="6.207042523s" totalImagesPullingTime="1.875780048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:36.206341287 +0000 UTC m=+60.694532493" watchObservedRunningTime="2026-07-21 14:33:36.207042523 +0000 UTC m=+60.695233729" Jul 21 14:33:36.215743 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.215707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:36.938336 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.938248 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:36.975724 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.975675 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-z24cv"] Jul 21 14:33:36.977950 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.977923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:36.982068 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.982043 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 14:33:36.982068 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.982062 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 14:33:36.982285 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.982253 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 14:33:36.982669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.982440 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fpthv\"" Jul 21 14:33:36.982669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.982470 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 14:33:36.982669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.982439 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 14:33:36.992297 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:36.992269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-z24cv"] Jul 21 14:33:37.010507 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.010472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df78n\" (UniqueName: \"kubernetes.io/projected/52ea19f8-d5ea-4ebd-a85a-7596f091f628-kube-api-access-df78n\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.010686 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.010538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.010686 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.010631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.010838 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.010696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52ea19f8-d5ea-4ebd-a85a-7596f091f628-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.012087 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.012058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:37.111373 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.111340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.111559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.111397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52ea19f8-d5ea-4ebd-a85a-7596f091f628-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.111559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.111468 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df78n\" (UniqueName: \"kubernetes.io/projected/52ea19f8-d5ea-4ebd-a85a-7596f091f628-kube-api-access-df78n\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.111666 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.111601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.111757 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:37.111739 2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 21 14:33:37.111820 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:37.111810 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-tls podName:52ea19f8-d5ea-4ebd-a85a-7596f091f628 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:37.61179349 +0000 UTC m=+62.099984674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-z24cv" (UID: "52ea19f8-d5ea-4ebd-a85a-7596f091f628") : secret "prometheus-operator-tls" not found Jul 21 14:33:37.112263 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.112213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52ea19f8-d5ea-4ebd-a85a-7596f091f628-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.114187 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.114163 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.120740 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.120712 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df78n\" (UniqueName: \"kubernetes.io/projected/52ea19f8-d5ea-4ebd-a85a-7596f091f628-kube-api-access-df78n\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.349200 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.349167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:37.525895 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.525862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:37.616375 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.616292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.618814 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.618789 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/52ea19f8-d5ea-4ebd-a85a-7596f091f628-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-z24cv\" (UID: \"52ea19f8-d5ea-4ebd-a85a-7596f091f628\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:37.625242 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.625192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:37.722990 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.722939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:37.822137 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.822107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:37.889498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:37.889420 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" Jul 21 14:33:38.101704 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:38.101678 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbwvc" Jul 21 14:33:38.216342 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:38.216257 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:38.363723 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:38.363680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:38.465974 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:38.465938 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:39.609886 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:39.609757 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-z24cv"] Jul 21 14:33:39.613449 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:39.613423 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ea19f8_d5ea_4ebd_a85a_7596f091f628.slice/crio-76b51d5ca8a0745072b99d43821965665dbc38ead480b73b5fd2ef91e5fd1b5b WatchSource:0}: Error finding container 76b51d5ca8a0745072b99d43821965665dbc38ead480b73b5fd2ef91e5fd1b5b: Status 404 returned error can't find the container with id 76b51d5ca8a0745072b99d43821965665dbc38ead480b73b5fd2ef91e5fd1b5b Jul 21 14:33:40.017374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.017343 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:40.119097 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.119061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:40.207239 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.207195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6695478675-mckkj" event={"ID":"9cb85d09-9d20-4aa3-8817-2f795fd49484","Type":"ContainerStarted","Data":"cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067"} Jul 21 14:33:40.209278 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.209246 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r9s6p" event={"ID":"4352a4a4-059c-433d-ba11-81053151750c","Type":"ContainerStarted","Data":"cd2f14f430ae374333ab0e39e0484fe0b9f0db89e29284c6d671f92a2792fbab"} Jul 21 14:33:40.210861 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.210839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846d44694f-5sjdx" event={"ID":"4f0631a7-8761-4ae0-9bc6-4b7813af7f71","Type":"ContainerStarted","Data":"7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7"} Jul 21 14:33:40.212057 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.212035 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" event={"ID":"52ea19f8-d5ea-4ebd-a85a-7596f091f628","Type":"ContainerStarted","Data":"76b51d5ca8a0745072b99d43821965665dbc38ead480b73b5fd2ef91e5fd1b5b"} Jul 21 14:33:40.228196 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.228155 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6695478675-mckkj" podStartSLOduration=2.610333996 podStartE2EDuration="8.22814109s" totalImagesPullingTime="5.617807094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:40.227541944 +0000 UTC m=+64.715733167" watchObservedRunningTime="2026-07-21 14:33:40.22814109 +0000 UTC m=+64.716332294" Jul 21 14:33:40.244349 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:40.244302 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-846d44694f-5sjdx" podStartSLOduration=4.602556684 podStartE2EDuration="10.244287942s" totalImagesPullingTime="5.641731258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:40.243889344 +0000 UTC m=+64.732080553" watchObservedRunningTime="2026-07-21 14:33:40.244287942 +0000 UTC m=+64.732479146" Jul 21 14:33:41.094313 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:41.094275 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:41.094757 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:41.094337 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:41.099996 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:41.099969 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:41.185610 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:41.185580 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tfp45" Jul 21 14:33:41.219382 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:41.219356 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:33:42.661253 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.661102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:42.661253 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.661177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:42.663974 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.663947 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 14:33:42.664070 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.664044 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 14:33:42.674346 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.674299 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41713217-eaa0-4eb2-b6f0-d89170d86af0-metrics-certs\") pod \"network-metrics-daemon-crw6b\" (UID: \"41713217-eaa0-4eb2-b6f0-d89170d86af0\") " pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:42.674436 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.674396 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92704e7d-742e-4358-a920-286027d064b0-original-pull-secret\") pod \"global-pull-secret-syncer-wpdjf\" (UID: \"92704e7d-742e-4358-a920-286027d064b0\") " pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:42.728121 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.728091 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cprh8\"" Jul 21 14:33:42.736584 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.736552 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crw6b" Jul 21 14:33:42.744617 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.744594 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpdjf" Jul 21 14:33:42.762488 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.762465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:42.765238 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.765203 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 14:33:42.776169 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.776147 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 14:33:42.787339 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.786756 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdpg\" (UniqueName: \"kubernetes.io/projected/74174e45-7725-4b5f-91d1-43ec79896c38-kube-api-access-nrdpg\") pod \"network-check-target-wlfd9\" (UID: \"74174e45-7725-4b5f-91d1-43ec79896c38\") " pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:42.798728 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:42.797410 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:43.039706 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.039669 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c99qd\"" Jul 21 14:33:43.046316 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.046286 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:43.086206 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.086173 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:43.086373 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.086263 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:43.091374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.091349 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:43.227522 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.227485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" event={"ID":"52ea19f8-d5ea-4ebd-a85a-7596f091f628","Type":"ContainerStarted","Data":"b1dc7a3adfb4b520bc3369e18d505dc9b407bb8a75ee70524c1c8af6df64e27e"} Jul 21 14:33:43.227673 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.227556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" event={"ID":"52ea19f8-d5ea-4ebd-a85a-7596f091f628","Type":"ContainerStarted","Data":"5917225608f5121a7590459c953aa7248c830158c2840b00e73a5108c347f42f"} Jul 21 14:33:43.231998 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.231977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:33:43.244772 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.244732 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-z24cv" podStartSLOduration=4.293755411 podStartE2EDuration="7.244718293s" totalImagesPullingTime="2.950962882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:43.243960026 +0000 UTC m=+67.732151261" watchObservedRunningTime="2026-07-21 14:33:43.244718293 +0000 UTC m=+67.732909497" Jul 21 14:33:43.291825 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.291756 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846d44694f-5sjdx"] Jul 21 14:33:43.569674 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.569599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:43.695895 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:43.695862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:44.548855 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:44.548628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlfd9"] Jul 21 14:33:44.552255 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:44.552231 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74174e45_7725_4b5f_91d1_43ec79896c38.slice/crio-74ad42d40db11e56867bd3372a270a0779359b40e55abfc640923ef36f954f03 WatchSource:0}: Error finding container 74ad42d40db11e56867bd3372a270a0779359b40e55abfc640923ef36f954f03: Status 404 returned error can't find the container with id 74ad42d40db11e56867bd3372a270a0779359b40e55abfc640923ef36f954f03 Jul 21 14:33:44.773236 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:44.773139 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wpdjf"] Jul 21 14:33:44.776300 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:44.776270 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crw6b"] Jul 21 14:33:44.776418 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:44.776316 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92704e7d_742e_4358_a920_286027d064b0.slice/crio-c3d0ba8187414cf6e612b2c94aec4e6edac204a8786920d04e9a6430ddcf8890 WatchSource:0}: Error finding container c3d0ba8187414cf6e612b2c94aec4e6edac204a8786920d04e9a6430ddcf8890: Status 404 returned error can't find the container with id c3d0ba8187414cf6e612b2c94aec4e6edac204a8786920d04e9a6430ddcf8890 Jul 21 14:33:44.779119 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:44.779096 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41713217_eaa0_4eb2_b6f0_d89170d86af0.slice/crio-22e521fc5baa75269bfe59fcf658ce538f6fba90cf0e1d878351556f173b331b WatchSource:0}: Error finding container 22e521fc5baa75269bfe59fcf658ce538f6fba90cf0e1d878351556f173b331b: Status 404 returned error can't find the container with id 22e521fc5baa75269bfe59fcf658ce538f6fba90cf0e1d878351556f173b331b Jul 21 14:33:44.809718 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:44.809692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:45.050383 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.050349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:45.142811 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.142776 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:45.240027 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.239995 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r9s6p" event={"ID":"4352a4a4-059c-433d-ba11-81053151750c","Type":"ContainerStarted","Data":"5605abc2caeab1b5bb746638a8fc8e6e07b6124337e30402088bfa2d9b1ac3e4"} Jul 21 14:33:45.241204 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.241179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlfd9" event={"ID":"74174e45-7725-4b5f-91d1-43ec79896c38","Type":"ContainerStarted","Data":"74ad42d40db11e56867bd3372a270a0779359b40e55abfc640923ef36f954f03"} Jul 21 14:33:45.242178 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.242146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crw6b" event={"ID":"41713217-eaa0-4eb2-b6f0-d89170d86af0","Type":"ContainerStarted","Data":"22e521fc5baa75269bfe59fcf658ce538f6fba90cf0e1d878351556f173b331b"} Jul 21 14:33:45.243059 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.243040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wpdjf" event={"ID":"92704e7d-742e-4358-a920-286027d064b0","Type":"ContainerStarted","Data":"c3d0ba8187414cf6e612b2c94aec4e6edac204a8786920d04e9a6430ddcf8890"} Jul 21 14:33:45.269070 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.269017 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r9s6p" podStartSLOduration=1.873386804 podStartE2EDuration="18.269000874s" totalImagesPullingTime="16.39561407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:33:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:45.267699508 +0000 UTC m=+69.755890741" watchObservedRunningTime="2026-07-21 14:33:45.269000874 +0000 UTC m=+69.757192075" Jul 21 14:33:45.332070 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.331984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:45.358146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.358113 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm"] Jul 21 14:33:45.363471 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.363440 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.366521 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.366498 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 21 14:33:45.366521 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.366513 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8w6j5\"" Jul 21 14:33:45.366916 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.366900 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cwqsw"] Jul 21 14:33:45.367067 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.367053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 21 14:33:45.369537 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.369514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.372344 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.372111 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b2phk\"" Jul 21 14:33:45.372344 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.372173 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 14:33:45.372344 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.372242 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 14:33:45.372596 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.372575 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 14:33:45.373953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.373933 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm"] Jul 21 14:33:45.391765 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.391740 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-765dl"] Jul 21 14:33:45.393887 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.393872 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.396189 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.396170 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 14:33:45.396305 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.396170 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rjp7v\"" Jul 21 14:33:45.396564 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.396524 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 14:33:45.396842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.396828 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 14:33:45.406775 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.406749 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-765dl"] Jul 21 14:33:45.434962 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.434918 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:45.484502 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7db794aa-bce8-4c7a-a0ec-32c49123856b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.484651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484504 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.484651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484529 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-root\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.484651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2pj\" (UniqueName: \"kubernetes.io/projected/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-api-access-zr2pj\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.484651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484599 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-sys\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e17f7b-b271-4f20-976b-b90048f33b92-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-textfile\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e8d538-3244-4033-a8d7-521344c52f96-metrics-client-ca\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-tls\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e17f7b-b271-4f20-976b-b90048f33b92-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484778 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-wtmp\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61e17f7b-b271-4f20-976b-b90048f33b92-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484904 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7db794aa-bce8-4c7a-a0ec-32c49123856b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.484940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.485205 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72zc7\" (UniqueName: \"kubernetes.io/projected/61e17f7b-b271-4f20-976b-b90048f33b92-kube-api-access-72zc7\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.485205 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.484980 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njm4w\" (UniqueName: \"kubernetes.io/projected/18e8d538-3244-4033-a8d7-521344c52f96-kube-api-access-njm4w\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586373 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2pj\" (UniqueName: \"kubernetes.io/projected/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-api-access-zr2pj\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.586373 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-sys\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586373 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586373 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e17f7b-b271-4f20-976b-b90048f33b92-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.586729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-sys\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-textfile\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586454 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e8d538-3244-4033-a8d7-521344c52f96-metrics-client-ca\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-tls\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e17f7b-b271-4f20-976b-b90048f33b92-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.586729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-wtmp\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.586729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-textfile\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61e17f7b-b271-4f20-976b-b90048f33b92-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586841 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7db794aa-bce8-4c7a-a0ec-32c49123856b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-wtmp\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72zc7\" (UniqueName: \"kubernetes.io/projected/61e17f7b-b271-4f20-976b-b90048f33b92-kube-api-access-72zc7\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.586970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njm4w\" (UniqueName: \"kubernetes.io/projected/18e8d538-3244-4033-a8d7-521344c52f96-kube-api-access-njm4w\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7db794aa-bce8-4c7a-a0ec-32c49123856b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-accelerators-collector-config\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587037 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.587073 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e8d538-3244-4033-a8d7-521344c52f96-metrics-client-ca\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-root\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.587669 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:45.587243 2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 21 14:33:45.587669 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:45.587316 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-tls podName:7db794aa-bce8-4c7a-a0ec-32c49123856b nodeName:}" failed. No retries permitted until 2026-07-21 14:33:46.087295844 +0000 UTC m=+70.575487047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-765dl" (UID: "7db794aa-bce8-4c7a-a0ec-32c49123856b") : secret "kube-state-metrics-tls" not found Jul 21 14:33:45.587669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18e8d538-3244-4033-a8d7-521344c52f96-root\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.587669 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.587447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61e17f7b-b271-4f20-976b-b90048f33b92-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.588262 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.588096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7db794aa-bce8-4c7a-a0ec-32c49123856b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.588262 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.588250 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7db794aa-bce8-4c7a-a0ec-32c49123856b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.588558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.588538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.589447 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.589422 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-tls\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.589547 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.589464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61e17f7b-b271-4f20-976b-b90048f33b92-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.589714 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.589696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61e17f7b-b271-4f20-976b-b90048f33b92-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.589850 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.589834 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.589890 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.589835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e8d538-3244-4033-a8d7-521344c52f96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.601518 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.601498 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2pj\" (UniqueName: \"kubernetes.io/projected/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-api-access-zr2pj\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:45.601622 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.601521 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njm4w\" (UniqueName: \"kubernetes.io/projected/18e8d538-3244-4033-a8d7-521344c52f96-kube-api-access-njm4w\") pod \"node-exporter-cwqsw\" (UID: \"18e8d538-3244-4033-a8d7-521344c52f96\") " pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.601842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.601824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72zc7\" (UniqueName: \"kubernetes.io/projected/61e17f7b-b271-4f20-976b-b90048f33b92-kube-api-access-72zc7\") pod \"openshift-state-metrics-79f6958cdf-gqjgm\" (UID: \"61e17f7b-b271-4f20-976b-b90048f33b92\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.673382 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.673354 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" Jul 21 14:33:45.680345 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.680320 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cwqsw" Jul 21 14:33:45.689414 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:45.689385 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e8d538_3244_4033_a8d7_521344c52f96.slice/crio-1dcfba55f1766c7d8d80ef4012c88d8b762710f5b806eb8d6cdada32453ea246 WatchSource:0}: Error finding container 1dcfba55f1766c7d8d80ef4012c88d8b762710f5b806eb8d6cdada32453ea246: Status 404 returned error can't find the container with id 1dcfba55f1766c7d8d80ef4012c88d8b762710f5b806eb8d6cdada32453ea246 Jul 21 14:33:45.797059 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:45.797037 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm"] Jul 21 14:33:45.799199 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:45.799178 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e17f7b_b271_4f20_976b_b90048f33b92.slice/crio-0298e05508ae88a99eee79a294d662b4b702b7d5bff59334e85d4b4f7d420ee7 WatchSource:0}: Error finding container 0298e05508ae88a99eee79a294d662b4b702b7d5bff59334e85d4b4f7d420ee7: Status 404 returned error can't find the container with id 0298e05508ae88a99eee79a294d662b4b702b7d5bff59334e85d4b4f7d420ee7 Jul 21 14:33:46.090928 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.090893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:46.093544 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.093515 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7db794aa-bce8-4c7a-a0ec-32c49123856b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-765dl\" (UID: \"7db794aa-bce8-4c7a-a0ec-32c49123856b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:46.249541 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.249451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" event={"ID":"61e17f7b-b271-4f20-976b-b90048f33b92","Type":"ContainerStarted","Data":"57397c3174b21009b3fcf2994b1d12715f7fe7494c059fd996fcf1a922bc8bbc"} Jul 21 14:33:46.249541 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.249533 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" event={"ID":"61e17f7b-b271-4f20-976b-b90048f33b92","Type":"ContainerStarted","Data":"881ed236245a5b6e37124a4f1f97fc4233617b23f958a9d916fcc8c27175f4ec"} Jul 21 14:33:46.249757 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.249563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" event={"ID":"61e17f7b-b271-4f20-976b-b90048f33b92","Type":"ContainerStarted","Data":"0298e05508ae88a99eee79a294d662b4b702b7d5bff59334e85d4b4f7d420ee7"} Jul 21 14:33:46.250912 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.250878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwqsw" event={"ID":"18e8d538-3244-4033-a8d7-521344c52f96","Type":"ContainerStarted","Data":"1dcfba55f1766c7d8d80ef4012c88d8b762710f5b806eb8d6cdada32453ea246"} Jul 21 14:33:46.307063 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.307034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" Jul 21 14:33:46.429874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.429446 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:33:46.441567 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.441528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.444126 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.444102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 21 14:33:46.444463 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.444441 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 21 14:33:46.444563 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.444474 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 21 14:33:46.444563 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.444476 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z5vlw\"" Jul 21 14:33:46.444963 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.444941 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 21 14:33:46.445053 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.444983 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 21 14:33:46.445053 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.444943 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 21 14:33:46.445326 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.445307 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 21 14:33:46.445501 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.445382 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 21 14:33:46.445882 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.445784 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 21 14:33:46.448449 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.448390 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:33:46.465010 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.464926 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-765dl"] Jul 21 14:33:46.467072 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:46.467048 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db794aa_bce8_4c7a_a0ec_32c49123856b.slice/crio-bf5ff20b0fbed23169da936ea3a512b06c65b4b615404f75d65af790077b29de WatchSource:0}: Error finding container bf5ff20b0fbed23169da936ea3a512b06c65b4b615404f75d65af790077b29de: Status 404 returned error can't find the container with id bf5ff20b0fbed23169da936ea3a512b06c65b4b615404f75d65af790077b29de Jul 21 14:33:46.493661 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.493624 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.493819 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.493694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-web-config\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.493819 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.493750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.493819 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.493779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.493981 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.493859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-out\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.493981 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.493902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-volume\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.493981 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.493927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.494121 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.494048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.494121 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.494094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.494211 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.494118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.494211 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.494165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.494211 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.494189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.494379 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.494234 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kntv\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-kube-api-access-5kntv\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.594988 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.594901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595291 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595256 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595431 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kntv\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-kube-api-access-5kntv\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595431 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595431 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-web-config\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595549 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595549 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595549 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-out\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595549 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595529 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-volume\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595732 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595732 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595732 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.595732 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.595667 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.596469 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.596404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.596566 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:46.596504 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle podName:eeded95d-e423-4ab5-b9a4-0e770f2b787b nodeName:}" failed. No retries permitted until 2026-07-21 14:33:47.096482021 +0000 UTC m=+71.584673233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b") : configmap references non-existent config key: ca-bundle.crt Jul 21 14:33:46.597203 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.597008 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.599944 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.599924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-volume\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.601097 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.600957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.601387 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.601300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-out\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.601480 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.601420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.602696 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.601907 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.602696 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.602345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.602852 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.602775 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-web-config\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.603075 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.603053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.604298 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.604276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.608646 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.608613 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kntv\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-kube-api-access-5kntv\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:46.895001 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.894872 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:46.966968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:46.966926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:47.100038 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.099997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:47.100852 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.100823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:47.258145 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.258108 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" event={"ID":"7db794aa-bce8-4c7a-a0ec-32c49123856b","Type":"ContainerStarted","Data":"bf5ff20b0fbed23169da936ea3a512b06c65b4b615404f75d65af790077b29de"} Jul 21 14:33:47.331940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.331910 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-ff49f758d-2f4qx"] Jul 21 14:33:47.347093 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.347062 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.351159 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.351121 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 14:33:47.351386 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.351362 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 14:33:47.352739 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.352080 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 14:33:47.352739 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.352358 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 14:33:47.352739 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.352574 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qrg9k\"" Jul 21 14:33:47.352940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.352763 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 14:33:47.352940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.352878 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-371git0qgb6ph\"" Jul 21 14:33:47.353714 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.353693 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:33:47.354756 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.354293 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ff49f758d-2f4qx"] Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.403726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-grpc-tls\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.403823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4fx\" (UniqueName: \"kubernetes.io/projected/a9f908db-11df-4eee-bc22-1dc0a79bb931-kube-api-access-2c4fx\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.403857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.403890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.403918 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-tls\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.403942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.403991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.404122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.404036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9f908db-11df-4eee-bc22-1dc0a79bb931-metrics-client-ca\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.506885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4fx\" (UniqueName: \"kubernetes.io/projected/a9f908db-11df-4eee-bc22-1dc0a79bb931-kube-api-access-2c4fx\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.506940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.506977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.507003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-tls\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.507029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.507059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.507100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9f908db-11df-4eee-bc22-1dc0a79bb931-metrics-client-ca\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.507968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.507151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-grpc-tls\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.509528 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.509478 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9f908db-11df-4eee-bc22-1dc0a79bb931-metrics-client-ca\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.514291 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.514011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.518597 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.518557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.529537 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.529022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-tls\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.532240 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.529990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.532240 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.530270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.532240 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.531578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9f908db-11df-4eee-bc22-1dc0a79bb931-secret-grpc-tls\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.532240 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.532047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4fx\" (UniqueName: \"kubernetes.io/projected/a9f908db-11df-4eee-bc22-1dc0a79bb931-kube-api-access-2c4fx\") pod \"thanos-querier-ff49f758d-2f4qx\" (UID: \"a9f908db-11df-4eee-bc22-1dc0a79bb931\") " pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.561688 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.561579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:33:47.587266 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:47.587205 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeded95d_e423_4ab5_b9a4_0e770f2b787b.slice/crio-5a9995e4c7d3ea18d42b7ceaccf58c375948423e3c06f5ef8ae176bc894e0847 WatchSource:0}: Error finding container 5a9995e4c7d3ea18d42b7ceaccf58c375948423e3c06f5ef8ae176bc894e0847: Status 404 returned error can't find the container with id 5a9995e4c7d3ea18d42b7ceaccf58c375948423e3c06f5ef8ae176bc894e0847 Jul 21 14:33:47.664803 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.664767 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:47.868566 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:47.868527 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-ff49f758d-2f4qx"] Jul 21 14:33:48.125062 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:48.125020 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f908db_11df_4eee_bc22_1dc0a79bb931.slice/crio-5f5e5d9d41f85141570e132ad596bd5646feeb0ad8d722bdd18ceb119f9365c9 WatchSource:0}: Error finding container 5f5e5d9d41f85141570e132ad596bd5646feeb0ad8d722bdd18ceb119f9365c9: Status 404 returned error can't find the container with id 5f5e5d9d41f85141570e132ad596bd5646feeb0ad8d722bdd18ceb119f9365c9 Jul 21 14:33:48.267089 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.267052 2559 generic.go:356] "Generic (PLEG): container finished" podID="18e8d538-3244-4033-a8d7-521344c52f96" containerID="04078ef353347e3e945ccff6f263bbd4614592395a6e73c7b05c4ec4f9fa2af1" exitCode=0 Jul 21 14:33:48.267236 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.267148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwqsw" event={"ID":"18e8d538-3244-4033-a8d7-521344c52f96","Type":"ContainerDied","Data":"04078ef353347e3e945ccff6f263bbd4614592395a6e73c7b05c4ec4f9fa2af1"} Jul 21 14:33:48.269914 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.269874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" event={"ID":"a9f908db-11df-4eee-bc22-1dc0a79bb931","Type":"ContainerStarted","Data":"5f5e5d9d41f85141570e132ad596bd5646feeb0ad8d722bdd18ceb119f9365c9"} Jul 21 14:33:48.272368 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.272190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerStarted","Data":"5a9995e4c7d3ea18d42b7ceaccf58c375948423e3c06f5ef8ae176bc894e0847"} Jul 21 14:33:48.316478 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.316443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:48.417732 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.417657 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:48.835940 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.835844 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:48.995325 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:48.995235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:49.283083 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.283040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crw6b" event={"ID":"41713217-eaa0-4eb2-b6f0-d89170d86af0","Type":"ContainerStarted","Data":"d52a7e4fbdc49e39a6dd3bb7b9e70db8be7a37b80b8398fd0cfe24620b700903"} Jul 21 14:33:49.752255 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.752156 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66b5c874f-8fb4d"] Jul 21 14:33:49.755412 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.755388 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.761112 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.761086 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2q9sj\"" Jul 21 14:33:49.761386 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.761356 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 14:33:49.761954 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.761555 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 14:33:49.761954 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.761694 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 14:33:49.761954 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.761810 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 14:33:49.761954 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.761914 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7u65srdvjusbj\"" Jul 21 14:33:49.766640 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.766614 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66b5c874f-8fb4d"] Jul 21 14:33:49.794108 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.794064 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:49.831445 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.831406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-metrics-server-audit-profiles\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.831623 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.831455 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-audit-log\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.831623 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.831510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.831623 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.831606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljh5r\" (UniqueName: \"kubernetes.io/projected/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-kube-api-access-ljh5r\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.831768 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.831663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-secret-metrics-server-client-certs\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.831768 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.831705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-secret-metrics-server-tls\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.831878 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.831771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-client-ca-bundle\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.851830 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.851791 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:49.932570 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.932538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljh5r\" (UniqueName: \"kubernetes.io/projected/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-kube-api-access-ljh5r\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.932738 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.932591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-secret-metrics-server-client-certs\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.932738 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.932620 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-secret-metrics-server-tls\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.932738 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.932675 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-client-ca-bundle\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.932738 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.932713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-metrics-server-audit-profiles\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.932970 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.932739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-audit-log\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.932970 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.932771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.933677 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.933612 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.934179 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.934154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-audit-log\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.934312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.934160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-metrics-server-audit-profiles\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.935604 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.935581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-secret-metrics-server-client-certs\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.935778 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.935760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-client-ca-bundle\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.935778 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.935768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-secret-metrics-server-tls\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:49.941393 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:49.941373 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljh5r\" (UniqueName: \"kubernetes.io/projected/9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42-kube-api-access-ljh5r\") pod \"metrics-server-66b5c874f-8fb4d\" (UID: \"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42\") " pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:50.070503 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.070469 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:33:50.125498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.125460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:50.129079 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.129048 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj"] Jul 21 14:33:50.132552 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.132532 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.134939 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.134915 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 14:33:50.134939 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.134931 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ptfhp\"" Jul 21 14:33:50.139920 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.139061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj"] Jul 21 14:33:50.229164 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.229124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:50.234628 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.234594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wngm\" (UniqueName: \"kubernetes.io/projected/733c0726-a082-4800-b7c1-4442623b4d99-kube-api-access-8wngm\") pod \"monitoring-plugin-7b7b7bc8bf-bqmlj\" (UID: \"733c0726-a082-4800-b7c1-4442623b4d99\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.234750 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.234640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/733c0726-a082-4800-b7c1-4442623b4d99-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-bqmlj\" (UID: \"733c0726-a082-4800-b7c1-4442623b4d99\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.335187 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.335109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wngm\" (UniqueName: \"kubernetes.io/projected/733c0726-a082-4800-b7c1-4442623b4d99-kube-api-access-8wngm\") pod \"monitoring-plugin-7b7b7bc8bf-bqmlj\" (UID: \"733c0726-a082-4800-b7c1-4442623b4d99\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.335187 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.335144 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/733c0726-a082-4800-b7c1-4442623b4d99-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-bqmlj\" (UID: \"733c0726-a082-4800-b7c1-4442623b4d99\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.335642 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:50.335294 2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 21 14:33:50.335642 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:33:50.335359 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733c0726-a082-4800-b7c1-4442623b4d99-monitoring-plugin-cert podName:733c0726-a082-4800-b7c1-4442623b4d99 nodeName:}" failed. No retries permitted until 2026-07-21 14:33:50.835341112 +0000 UTC m=+75.323532294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/733c0726-a082-4800-b7c1-4442623b4d99-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-bqmlj" (UID: "733c0726-a082-4800-b7c1-4442623b4d99") : secret "monitoring-plugin-cert" not found Jul 21 14:33:50.347519 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.347490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wngm\" (UniqueName: \"kubernetes.io/projected/733c0726-a082-4800-b7c1-4442623b4d99-kube-api-access-8wngm\") pod \"monitoring-plugin-7b7b7bc8bf-bqmlj\" (UID: \"733c0726-a082-4800-b7c1-4442623b4d99\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.419364 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.419330 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:50.610951 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.610865 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2"] Jul 21 14:33:50.615543 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.615522 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.618391 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.618329 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 21 14:33:50.618391 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.618341 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-45dbw\"" Jul 21 14:33:50.618391 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.618369 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 21 14:33:50.618391 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.618385 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 21 14:33:50.618657 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.618371 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 21 14:33:50.618657 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.618411 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 21 14:33:50.623738 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.623716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2"] Jul 21 14:33:50.625237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.625197 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 21 14:33:50.739046 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-telemeter-client-tls\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.739227 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.739227 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-serving-certs-ca-bundle\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.739227 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-metrics-client-ca\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.739227 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.739482 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-secret-telemeter-client\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.739482 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-federate-client-tls\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.739482 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.739449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z777t\" (UniqueName: \"kubernetes.io/projected/3b08755e-797d-4094-8bf9-e0f78356bea6-kube-api-access-z777t\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840092 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-secret-telemeter-client\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840092 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/733c0726-a082-4800-b7c1-4442623b4d99-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-bqmlj\" (UID: \"733c0726-a082-4800-b7c1-4442623b4d99\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.840347 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-federate-client-tls\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840347 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z777t\" (UniqueName: \"kubernetes.io/projected/3b08755e-797d-4094-8bf9-e0f78356bea6-kube-api-access-z777t\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840347 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-telemeter-client-tls\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840347 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-serving-certs-ca-bundle\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-metrics-client-ca\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.840559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.840446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.841402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.841323 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-serving-certs-ca-bundle\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.841507 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.841431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.841766 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.841716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b08755e-797d-4094-8bf9-e0f78356bea6-metrics-client-ca\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.843183 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.843153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-federate-client-tls\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.843327 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.843308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-telemeter-client-tls\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.843393 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.843350 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.843804 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.843779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3b08755e-797d-4094-8bf9-e0f78356bea6-secret-telemeter-client\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.843922 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.843824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/733c0726-a082-4800-b7c1-4442623b4d99-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-bqmlj\" (UID: \"733c0726-a082-4800-b7c1-4442623b4d99\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:50.850790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.850772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z777t\" (UniqueName: \"kubernetes.io/projected/3b08755e-797d-4094-8bf9-e0f78356bea6-kube-api-access-z777t\") pod \"telemeter-client-ccb7d6cb4-rlnd2\" (UID: \"3b08755e-797d-4094-8bf9-e0f78356bea6\") " pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.928188 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.928121 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" Jul 21 14:33:50.967671 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.967636 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:50.975364 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.975319 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b7cf449b5-68bxz"] Jul 21 14:33:50.979016 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.978995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:50.989003 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:50.988954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b7cf449b5-68bxz"] Jul 21 14:33:51.042067 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.042036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-oauth-serving-cert\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.042067 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.042067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-trusted-ca-bundle\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.042318 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.042121 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-config\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.042318 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.042200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-serving-cert\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.042318 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.042261 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-service-ca\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.042318 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.042309 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7zv\" (UniqueName: \"kubernetes.io/projected/93e17f1e-700c-49a0-b90f-fc8d8571801b-kube-api-access-jh7zv\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.042506 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.042329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-oauth-config\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.046942 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.046913 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:51.143339 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.143308 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-service-ca\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.143496 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.143370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7zv\" (UniqueName: \"kubernetes.io/projected/93e17f1e-700c-49a0-b90f-fc8d8571801b-kube-api-access-jh7zv\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.143496 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.143402 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-oauth-config\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.143496 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.143442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-oauth-serving-cert\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.143496 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.143466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-trusted-ca-bundle\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.143698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.143513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-config\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.143698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.143563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-serving-cert\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.144211 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.144160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-service-ca\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.144449 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.144397 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-config\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.144449 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.144424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-oauth-serving-cert\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.144614 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.144459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-trusted-ca-bundle\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.146210 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.146192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-oauth-config\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.146462 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.146440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-serving-cert\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.152184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.152160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7zv\" (UniqueName: \"kubernetes.io/projected/93e17f1e-700c-49a0-b90f-fc8d8571801b-kube-api-access-jh7zv\") pod \"console-6b7cf449b5-68bxz\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.291342 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.291315 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:33:51.565257 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.565102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:51.602229 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.602192 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:33:51.607127 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.607101 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.613265 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.613240 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 14:33:51.613803 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.613780 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 14:33:51.616076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.616050 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 14:33:51.616193 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.616092 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 14:33:51.616309 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.616262 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 14:33:51.616932 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.616786 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fbr435ta36n1k\"" Jul 21 14:33:51.617238 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617151 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 14:33:51.617238 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617178 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 14:33:51.617683 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617398 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 14:33:51.617683 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617437 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 14:33:51.617683 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617460 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h5k9q\"" Jul 21 14:33:51.617683 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617472 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 14:33:51.617683 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617397 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 14:33:51.617683 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.617653 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 14:33:51.621886 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.621272 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 14:33:51.624058 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.624034 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 14:33:51.630435 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.630412 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:33:51.685680 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.685603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:51.749555 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-config-out\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749695 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749695 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2mr\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-kube-api-access-pd2mr\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749695 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749695 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749919 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-config\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.749953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749940 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750092 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749960 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750092 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.749999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750092 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.750055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750189 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.750154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750189 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.750180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750279 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.750208 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750279 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.750252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750279 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.750297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-web-config\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.750433 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.750398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.851726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.851726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-config-out\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.851726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.851726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851676 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2mr\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-kube-api-access-pd2mr\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.851726 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851733 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.851835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852032 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-config\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852120 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852326 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-web-config\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.853338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.852620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.855929 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.854758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.855929 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.855356 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.855929 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.855654 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.856328 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.856306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.857086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.857053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.858542 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.858514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.859078 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.859023 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-config\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.859307 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.859261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-config-out\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.860175 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.860157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.861035 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.860993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.861525 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.861501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.861618 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.861511 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.861935 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.861914 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-web-config\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.862017 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.861962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.863509 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.863462 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.864293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.864242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.864481 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.864435 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.866005 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.865979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.866398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.866364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2mr\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-kube-api-access-pd2mr\") pod \"prometheus-k8s-0\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:51.921731 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:51.921690 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:33:52.423561 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:52.423526 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b7cf449b5-68bxz"] Jul 21 14:33:52.579963 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:52.579935 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e17f1e_700c_49a0_b90f_fc8d8571801b.slice/crio-fdff12ad5f17d2fd396e2266eb694fe553c6150460f1ea974a65b35649560745 WatchSource:0}: Error finding container fdff12ad5f17d2fd396e2266eb694fe553c6150460f1ea974a65b35649560745: Status 404 returned error can't find the container with id fdff12ad5f17d2fd396e2266eb694fe553c6150460f1ea974a65b35649560745 Jul 21 14:33:52.795630 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:52.785005 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2"] Jul 21 14:33:52.827748 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:52.827636 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66b5c874f-8fb4d"] Jul 21 14:33:52.839651 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:52.839590 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4d2521_bed9_4fbc_bf84_dbca7d7c5c42.slice/crio-fa763684f59a575bfca967da96f47f9047e7dda73b5f8436971c135f4edc1b5c WatchSource:0}: Error finding container fa763684f59a575bfca967da96f47f9047e7dda73b5f8436971c135f4edc1b5c: Status 404 returned error can't find the container with id fa763684f59a575bfca967da96f47f9047e7dda73b5f8436971c135f4edc1b5c Jul 21 14:33:53.086272 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.086245 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj"] Jul 21 14:33:53.087871 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:53.087844 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733c0726_a082_4800_b7c1_4442623b4d99.slice/crio-f81d5f59381ada650f97dda714bf31981d7e708a68ecf5f6705ca1c44facafd9 WatchSource:0}: Error finding container f81d5f59381ada650f97dda714bf31981d7e708a68ecf5f6705ca1c44facafd9: Status 404 returned error can't find the container with id f81d5f59381ada650f97dda714bf31981d7e708a68ecf5f6705ca1c44facafd9 Jul 21 14:33:53.101402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.099443 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:33:53.105550 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:33:53.105058 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18441986_b4c9_40ed_9090_51080bca018f.slice/crio-e2e9228c0138b876c8932a3d47b57525ac054191b9c60dc201eb385a4b9c88f8 WatchSource:0}: Error finding container e2e9228c0138b876c8932a3d47b57525ac054191b9c60dc201eb385a4b9c88f8: Status 404 returned error can't find the container with id e2e9228c0138b876c8932a3d47b57525ac054191b9c60dc201eb385a4b9c88f8 Jul 21 14:33:53.148961 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.148933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:53.269639 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.269608 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:53.302362 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.302332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlfd9" event={"ID":"74174e45-7725-4b5f-91d1-43ec79896c38","Type":"ContainerStarted","Data":"2c883a124a8ec4e923ff2898dc4344d26816bf7b15cde563b46a2769dad73a78"} Jul 21 14:33:53.302548 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.302402 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:33:53.304805 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.304780 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwqsw" event={"ID":"18e8d538-3244-4033-a8d7-521344c52f96","Type":"ContainerStarted","Data":"fb6e366b60a9eb875cbd9bfd255c54aaef68d97db8e68ecb2856ef24e19f63ed"} Jul 21 14:33:53.304919 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.304830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cwqsw" event={"ID":"18e8d538-3244-4033-a8d7-521344c52f96","Type":"ContainerStarted","Data":"be672b0bd20e28a03ac9a4bf36cba13b4de688dae7525a3838f8ebf7230e2e66"} Jul 21 14:33:53.307072 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.307044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" event={"ID":"a9f908db-11df-4eee-bc22-1dc0a79bb931","Type":"ContainerStarted","Data":"62b5de4d3d31421a7af2e2319b3056eab7661b4edbd757c5e6a0ac798c70824c"} Jul 21 14:33:53.307241 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.307097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" event={"ID":"a9f908db-11df-4eee-bc22-1dc0a79bb931","Type":"ContainerStarted","Data":"e49932f93a2c57da4ff0869cbfb81fec87a5b9ea68ee8ec7f49f1448ecc7e4bc"} Jul 21 14:33:53.307241 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.307140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" event={"ID":"a9f908db-11df-4eee-bc22-1dc0a79bb931","Type":"ContainerStarted","Data":"c1c3d67cce7aca27ae31d95ff6b3bd071951d6ad5b04bf41385393a0cda78994"} Jul 21 14:33:53.308634 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.308612 2559 generic.go:356] "Generic (PLEG): container finished" podID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerID="c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1" exitCode=0 Jul 21 14:33:53.308734 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.308696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1"} Jul 21 14:33:53.310950 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.310924 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crw6b" event={"ID":"41713217-eaa0-4eb2-b6f0-d89170d86af0","Type":"ContainerStarted","Data":"c165ec2a6a304e3951bfb4eaf5af7c9845657b55b86c32bf5dfcd42d1ffc1423"} Jul 21 14:33:53.313502 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.313458 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cf449b5-68bxz" event={"ID":"93e17f1e-700c-49a0-b90f-fc8d8571801b","Type":"ContainerStarted","Data":"587eb102df1b5d802b5c5cfcafb3a9190c82f5c39f63486d2ccc3c2e67f1b849"} Jul 21 14:33:53.313502 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.313502 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cf449b5-68bxz" event={"ID":"93e17f1e-700c-49a0-b90f-fc8d8571801b","Type":"ContainerStarted","Data":"fdff12ad5f17d2fd396e2266eb694fe553c6150460f1ea974a65b35649560745"} Jul 21 14:33:53.315183 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.315148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wpdjf" event={"ID":"92704e7d-742e-4358-a920-286027d064b0","Type":"ContainerStarted","Data":"cc36a9bed0167cea08b0736c60cff9e22e64b1e3073c9d5d13715494fdc56b86"} Jul 21 14:33:53.316619 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.316583 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" event={"ID":"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42","Type":"ContainerStarted","Data":"fa763684f59a575bfca967da96f47f9047e7dda73b5f8436971c135f4edc1b5c"} Jul 21 14:33:53.319101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.318435 2559 generic.go:356] "Generic (PLEG): container finished" podID="18441986-b4c9-40ed-9090-51080bca018f" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" exitCode=0 Jul 21 14:33:53.319101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.318512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43"} Jul 21 14:33:53.319101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.318569 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerStarted","Data":"e2e9228c0138b876c8932a3d47b57525ac054191b9c60dc201eb385a4b9c88f8"} Jul 21 14:33:53.319794 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.319675 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wlfd9" podStartSLOduration=68.271304556 podStartE2EDuration="1m16.319657338s" totalImagesPullingTime="8.048352782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:53.318107928 +0000 UTC m=+77.806299155" watchObservedRunningTime="2026-07-21 14:33:53.319657338 +0000 UTC m=+77.807848544" Jul 21 14:33:53.321152 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.321120 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" event={"ID":"7db794aa-bce8-4c7a-a0ec-32c49123856b","Type":"ContainerStarted","Data":"f867a22ac3bcf1ad11f3a7dca23a9569ed8be418ec8bac01e488ad0225469973"} Jul 21 14:33:53.321256 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.321168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" event={"ID":"7db794aa-bce8-4c7a-a0ec-32c49123856b","Type":"ContainerStarted","Data":"ef5a3c7160fe904e50a8a921e058d5f660fb372fa25f2be0c75eda43dd4aa891"} Jul 21 14:33:53.321256 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.321190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" event={"ID":"7db794aa-bce8-4c7a-a0ec-32c49123856b","Type":"ContainerStarted","Data":"a2ef5ece6305d3283b6a6a79e38ec119b73ff4c6a795c6dde87fe6dd4e98110d"} Jul 21 14:33:53.322505 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.322478 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" event={"ID":"733c0726-a082-4800-b7c1-4442623b4d99","Type":"ContainerStarted","Data":"f81d5f59381ada650f97dda714bf31981d7e708a68ecf5f6705ca1c44facafd9"} Jul 21 14:33:53.323756 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.323696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" event={"ID":"3b08755e-797d-4094-8bf9-e0f78356bea6","Type":"ContainerStarted","Data":"ece0ff4e6c08d1b3aa9944ab3eb41243841936d80b4a5905db73fc6df11db572"} Jul 21 14:33:53.326072 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.326052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" event={"ID":"61e17f7b-b271-4f20-976b-b90048f33b92","Type":"ContainerStarted","Data":"56f71a8c17b3ffcffb5d6842185e7584a74540b6304a1d7268dd24c8af623243"} Jul 21 14:33:53.361095 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.360908 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cwqsw" podStartSLOduration=6.688923878 podStartE2EDuration="8.360887349s" totalImagesPullingTime="1.531216685s" totalInitContainerRuntime="140.746786ms" isStatefulPod=true podCreationTimestamp="2026-07-21 14:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:53.36060662 +0000 UTC m=+77.848797900" watchObservedRunningTime="2026-07-21 14:33:53.360887349 +0000 UTC m=+77.849078544" Jul 21 14:33:53.375874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.375817 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wpdjf" podStartSLOduration=68.551027941 podStartE2EDuration="1m16.375799359s" totalImagesPullingTime="7.824771418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:53.375105808 +0000 UTC m=+77.863297014" watchObservedRunningTime="2026-07-21 14:33:53.375799359 +0000 UTC m=+77.863990565" Jul 21 14:33:53.392617 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.392550 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-crw6b" podStartSLOduration=73.016954345 podStartE2EDuration="1m16.392526011s" totalImagesPullingTime="3.375571666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:53.390194144 +0000 UTC m=+77.878385383" watchObservedRunningTime="2026-07-21 14:33:53.392526011 +0000 UTC m=+77.880717215" Jul 21 14:33:53.407978 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.407914 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b7cf449b5-68bxz" podStartSLOduration=3.407896816 podStartE2EDuration="3.407896816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:53.406955499 +0000 UTC m=+77.895146726" watchObservedRunningTime="2026-07-21 14:33:53.407896816 +0000 UTC m=+77.896087997" Jul 21 14:33:53.426772 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.426717 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-765dl" podStartSLOduration=2.604542097 podStartE2EDuration="8.426698898s" totalImagesPullingTime="5.822156801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:53.424794401 +0000 UTC m=+77.912985638" watchObservedRunningTime="2026-07-21 14:33:53.426698898 +0000 UTC m=+77.914890102" Jul 21 14:33:53.445376 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.445313 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gqjgm" podStartSLOduration=2.079381407 podStartE2EDuration="8.44529175s" totalImagesPullingTime="6.365910343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:53.442751435 +0000 UTC m=+77.930942640" watchObservedRunningTime="2026-07-21 14:33:53.44529175 +0000 UTC m=+77.933482953" Jul 21 14:33:53.561521 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.561484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:53.661666 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:53.661591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:55.345458 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:55.345416 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" event={"ID":"a9f908db-11df-4eee-bc22-1dc0a79bb931","Type":"ContainerStarted","Data":"22c596e05e5ba1364e9842854beb19386b24799f573280c86bfe17287c9a9a50"} Jul 21 14:33:55.345917 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:55.345513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" event={"ID":"a9f908db-11df-4eee-bc22-1dc0a79bb931","Type":"ContainerStarted","Data":"9c636f2f76c14a7bc1e27dbfd21793930919c6ee0e76e6209f04353b3d8450e0"} Jul 21 14:33:55.348673 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:55.348641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" event={"ID":"9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42","Type":"ContainerStarted","Data":"1957c1d0087bd0b156c27fd5c84e64e2a48b8a34c13d88f19dd259e25c76a712"} Jul 21 14:33:55.371995 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:55.370561 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" podStartSLOduration=4.580670047 podStartE2EDuration="6.370541757s" totalImagesPullingTime="1.78987171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:55.369743452 +0000 UTC m=+79.857934675" watchObservedRunningTime="2026-07-21 14:33:55.370541757 +0000 UTC m=+79.858732970" Jul 21 14:33:56.356536 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:56.356441 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" event={"ID":"a9f908db-11df-4eee-bc22-1dc0a79bb931","Type":"ContainerStarted","Data":"b6e2ed4bbf78cd4370b7004869c91027a93c6ebd7bb05a2808559722f8e55ea4"} Jul 21 14:33:56.389503 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:56.389447 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" podStartSLOduration=3.354955975 podStartE2EDuration="9.389427751s" totalImagesPullingTime="6.034471776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:56.388007939 +0000 UTC m=+80.876199144" watchObservedRunningTime="2026-07-21 14:33:56.389427751 +0000 UTC m=+80.877618954" Jul 21 14:33:57.361736 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:57.361703 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:58.370061 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.370019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerStarted","Data":"bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08"} Jul 21 14:33:58.370558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.370138 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerStarted","Data":"4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c"} Jul 21 14:33:58.370558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.370189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerStarted","Data":"a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d"} Jul 21 14:33:58.370558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.370245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerStarted","Data":"a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89"} Jul 21 14:33:58.370558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.370275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerStarted","Data":"8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf"} Jul 21 14:33:58.370558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.370316 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerStarted","Data":"592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09"} Jul 21 14:33:58.373340 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.373313 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerStarted","Data":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} Jul 21 14:33:58.373486 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.373373 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerStarted","Data":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} Jul 21 14:33:58.373486 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.373412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerStarted","Data":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} Jul 21 14:33:58.373486 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.373441 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerStarted","Data":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} Jul 21 14:33:58.373486 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.373464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerStarted","Data":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} Jul 21 14:33:58.373668 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.373499 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerStarted","Data":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} Jul 21 14:33:58.374854 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.374828 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" event={"ID":"733c0726-a082-4800-b7c1-4442623b4d99","Type":"ContainerStarted","Data":"72e5e39b3be50ba01435ef36f6a349a694acd2f68dbc0aa1ce83ceb25e92743b"} Jul 21 14:33:58.374971 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.374956 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:58.377129 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.377107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" event={"ID":"3b08755e-797d-4094-8bf9-e0f78356bea6","Type":"ContainerStarted","Data":"35346e1699f02eeee4d827eaaf230655a4e462b560bb12b7fa584db5ac3fb686"} Jul 21 14:33:58.377265 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.377152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" event={"ID":"3b08755e-797d-4094-8bf9-e0f78356bea6","Type":"ContainerStarted","Data":"73b5ccfd928a936c260d2cf867b9885c2e8a8f821d00c222914bc6a357f24eb9"} Jul 21 14:33:58.377265 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.377170 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" event={"ID":"3b08755e-797d-4094-8bf9-e0f78356bea6","Type":"ContainerStarted","Data":"ee96a3f1c5f874cf8f3deb8fde666525ac7cb5cd566a4e26c73332da2dfd40fa"} Jul 21 14:33:58.380308 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.380288 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" Jul 21 14:33:58.383693 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.383676 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-ff49f758d-2f4qx" Jul 21 14:33:58.403552 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.403508 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9523953389999997 podStartE2EDuration="12.403492592s" totalImagesPullingTime="9.413874032s" totalInitContainerRuntime="37.223221ms" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:58.401956772 +0000 UTC m=+82.890147996" watchObservedRunningTime="2026-07-21 14:33:58.403492592 +0000 UTC m=+82.891683841" Jul 21 14:33:58.427176 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.427049 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-ccb7d6cb4-rlnd2" podStartSLOduration=3.542670214 podStartE2EDuration="8.427029023s" totalImagesPullingTime="4.884358809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:58.426077411 +0000 UTC m=+82.914268616" watchObservedRunningTime="2026-07-21 14:33:58.427029023 +0000 UTC m=+82.915220228" Jul 21 14:33:58.460234 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:58.460156 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.071692167 podStartE2EDuration="7.460135135s" totalImagesPullingTime="4.369325105s" totalInitContainerRuntime="19.117863ms" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:58.459425477 +0000 UTC m=+82.947616706" watchObservedRunningTime="2026-07-21 14:33:58.460135135 +0000 UTC m=+82.948326339" Jul 21 14:33:59.232660 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:59.232602 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bqmlj" podStartSLOduration=4.645614725 podStartE2EDuration="9.232586353s" totalImagesPullingTime="4.586971628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:58.519941346 +0000 UTC m=+83.008132551" watchObservedRunningTime="2026-07-21 14:33:59.232586353 +0000 UTC m=+83.720777556" Jul 21 14:33:59.232902 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:59.232797 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b7cf449b5-68bxz"] Jul 21 14:33:59.423512 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:59.423479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:59.524703 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:33:59.524615 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:01.292444 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:01.292387 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:34:01.922825 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:01.922793 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:34:05.104029 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:05.103990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:05.204465 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:05.204419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:09.261184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:09.261099 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-846d44694f-5sjdx" podUID="4f0631a7-8761-4ae0-9bc6-4b7813af7f71" containerName="console" containerID="cri-o://7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7" gracePeriod=15 Jul 21 14:34:10.066883 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.066851 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:34:10.071045 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.071022 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:34:10.071187 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.071068 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:34:10.223729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.223633 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjkl6\" (UniqueName: \"kubernetes.io/projected/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-kube-api-access-tjkl6\") pod \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " Jul 21 14:34:10.223729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.223675 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-config\") pod \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " Jul 21 14:34:10.223729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.223703 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-serving-cert\") pod \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " Jul 21 14:34:10.223729 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.223723 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-service-ca\") pod \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " Jul 21 14:34:10.224044 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.223758 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-oauth-config\") pod \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " Jul 21 14:34:10.224044 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.223779 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-oauth-serving-cert\") pod \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\" (UID: \"4f0631a7-8761-4ae0-9bc6-4b7813af7f71\") " Jul 21 14:34:10.224273 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.224244 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4f0631a7-8761-4ae0-9bc6-4b7813af7f71" (UID: "4f0631a7-8761-4ae0-9bc6-4b7813af7f71"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:10.224380 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.224150 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-service-ca" (OuterVolumeSpecName: "service-ca") pod "4f0631a7-8761-4ae0-9bc6-4b7813af7f71" (UID: "4f0631a7-8761-4ae0-9bc6-4b7813af7f71"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:10.224380 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.224304 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-config" (OuterVolumeSpecName: "console-config") pod "4f0631a7-8761-4ae0-9bc6-4b7813af7f71" (UID: "4f0631a7-8761-4ae0-9bc6-4b7813af7f71"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:10.226239 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.226177 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4f0631a7-8761-4ae0-9bc6-4b7813af7f71" (UID: "4f0631a7-8761-4ae0-9bc6-4b7813af7f71"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:34:10.226239 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.226199 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-kube-api-access-tjkl6" (OuterVolumeSpecName: "kube-api-access-tjkl6") pod "4f0631a7-8761-4ae0-9bc6-4b7813af7f71" (UID: "4f0631a7-8761-4ae0-9bc6-4b7813af7f71"). InnerVolumeSpecName "kube-api-access-tjkl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:34:10.226379 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.226258 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4f0631a7-8761-4ae0-9bc6-4b7813af7f71" (UID: "4f0631a7-8761-4ae0-9bc6-4b7813af7f71"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:34:10.324542 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.324503 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkl6\" (UniqueName: \"kubernetes.io/projected/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-kube-api-access-tjkl6\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:10.324542 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.324537 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:10.324542 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.324550 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-serving-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:10.324978 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.324562 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:10.324978 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.324574 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-console-oauth-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:10.324978 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.324584 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f0631a7-8761-4ae0-9bc6-4b7813af7f71-oauth-serving-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:10.428139 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.428104 2559 generic.go:356] "Generic (PLEG): container finished" podID="4f0631a7-8761-4ae0-9bc6-4b7813af7f71" containerID="7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7" exitCode=0 Jul 21 14:34:10.428312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.428153 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846d44694f-5sjdx" Jul 21 14:34:10.428312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.428185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846d44694f-5sjdx" event={"ID":"4f0631a7-8761-4ae0-9bc6-4b7813af7f71","Type":"ContainerDied","Data":"7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7"} Jul 21 14:34:10.428312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.428273 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846d44694f-5sjdx" event={"ID":"4f0631a7-8761-4ae0-9bc6-4b7813af7f71","Type":"ContainerDied","Data":"fce986bcc4eb68cea024c4707376e7656fcd343e0da43c08fb3848878051c2e0"} Jul 21 14:34:10.428312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.428300 2559 scope.go:117] "RemoveContainer" containerID="7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7" Jul 21 14:34:10.446022 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.446004 2559 scope.go:117] "RemoveContainer" containerID="7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7" Jul 21 14:34:10.446408 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:34:10.446364 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7\": container with ID starting with 7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7 not found: ID does not exist" containerID="7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7" Jul 21 14:34:10.446508 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.446404 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7"} err="failed to get container status \"7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7\": rpc error: code = NotFound desc = could not find container \"7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7\": container with ID starting with 7133502aceaf86d1a10ae3ad5829530c8e99a07b4a3ac721624ecaf98ef752f7 not found: ID does not exist" Jul 21 14:34:10.448408 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.448389 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846d44694f-5sjdx"] Jul 21 14:34:10.452379 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.452360 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-846d44694f-5sjdx"] Jul 21 14:34:10.644443 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.644412 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6695478675-mckkj"] Jul 21 14:34:10.789051 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.789018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:10.888943 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.888908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:10.915063 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.914994 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f0631a7-8761-4ae0-9bc6-4b7813af7f71" path="/var/lib/kubelet/pods/4f0631a7-8761-4ae0-9bc6-4b7813af7f71/volumes" Jul 21 14:34:10.943100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:10.943076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:18.555123 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:18.555071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:18.619975 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:18.619932 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:20.100535 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:20.100497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:20.158448 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:20.158418 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:24.253044 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.252998 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b7cf449b5-68bxz" podUID="93e17f1e-700c-49a0-b90f-fc8d8571801b" containerName="console" containerID="cri-o://587eb102df1b5d802b5c5cfcafb3a9190c82f5c39f63486d2ccc3c2e67f1b849" gracePeriod=15 Jul 21 14:34:24.336015 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.335986 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wlfd9" Jul 21 14:34:24.496158 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.496127 2559 generic.go:356] "Generic (PLEG): container finished" podID="93e17f1e-700c-49a0-b90f-fc8d8571801b" containerID="587eb102df1b5d802b5c5cfcafb3a9190c82f5c39f63486d2ccc3c2e67f1b849" exitCode=0 Jul 21 14:34:24.496312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.496183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cf449b5-68bxz" event={"ID":"93e17f1e-700c-49a0-b90f-fc8d8571801b","Type":"ContainerDied","Data":"587eb102df1b5d802b5c5cfcafb3a9190c82f5c39f63486d2ccc3c2e67f1b849"} Jul 21 14:34:24.496312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.496236 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7cf449b5-68bxz" event={"ID":"93e17f1e-700c-49a0-b90f-fc8d8571801b","Type":"ContainerDied","Data":"fdff12ad5f17d2fd396e2266eb694fe553c6150460f1ea974a65b35649560745"} Jul 21 14:34:24.496312 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.496255 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdff12ad5f17d2fd396e2266eb694fe553c6150460f1ea974a65b35649560745" Jul 21 14:34:24.497125 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.497108 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:34:24.559695 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.559663 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-config\") pod \"93e17f1e-700c-49a0-b90f-fc8d8571801b\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " Jul 21 14:34:24.559874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.559702 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-service-ca\") pod \"93e17f1e-700c-49a0-b90f-fc8d8571801b\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " Jul 21 14:34:24.559874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.559843 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-oauth-serving-cert\") pod \"93e17f1e-700c-49a0-b90f-fc8d8571801b\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " Jul 21 14:34:24.559983 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.559874 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh7zv\" (UniqueName: \"kubernetes.io/projected/93e17f1e-700c-49a0-b90f-fc8d8571801b-kube-api-access-jh7zv\") pod \"93e17f1e-700c-49a0-b90f-fc8d8571801b\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " Jul 21 14:34:24.559983 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.559936 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-trusted-ca-bundle\") pod \"93e17f1e-700c-49a0-b90f-fc8d8571801b\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " Jul 21 14:34:24.559983 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.559960 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-serving-cert\") pod \"93e17f1e-700c-49a0-b90f-fc8d8571801b\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " Jul 21 14:34:24.560123 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.559991 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-oauth-config\") pod \"93e17f1e-700c-49a0-b90f-fc8d8571801b\" (UID: \"93e17f1e-700c-49a0-b90f-fc8d8571801b\") " Jul 21 14:34:24.560197 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.560080 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-config" (OuterVolumeSpecName: "console-config") pod "93e17f1e-700c-49a0-b90f-fc8d8571801b" (UID: "93e17f1e-700c-49a0-b90f-fc8d8571801b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:24.560197 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.560185 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-service-ca" (OuterVolumeSpecName: "service-ca") pod "93e17f1e-700c-49a0-b90f-fc8d8571801b" (UID: "93e17f1e-700c-49a0-b90f-fc8d8571801b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:24.560336 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.560321 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:24.560394 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.560343 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:24.560441 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.560304 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "93e17f1e-700c-49a0-b90f-fc8d8571801b" (UID: "93e17f1e-700c-49a0-b90f-fc8d8571801b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:24.560492 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.560463 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "93e17f1e-700c-49a0-b90f-fc8d8571801b" (UID: "93e17f1e-700c-49a0-b90f-fc8d8571801b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:24.562265 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.562228 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "93e17f1e-700c-49a0-b90f-fc8d8571801b" (UID: "93e17f1e-700c-49a0-b90f-fc8d8571801b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:34:24.562265 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.562259 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "93e17f1e-700c-49a0-b90f-fc8d8571801b" (UID: "93e17f1e-700c-49a0-b90f-fc8d8571801b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:34:24.562405 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.562265 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e17f1e-700c-49a0-b90f-fc8d8571801b-kube-api-access-jh7zv" (OuterVolumeSpecName: "kube-api-access-jh7zv") pod "93e17f1e-700c-49a0-b90f-fc8d8571801b" (UID: "93e17f1e-700c-49a0-b90f-fc8d8571801b"). InnerVolumeSpecName "kube-api-access-jh7zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:34:24.660810 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.660766 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh7zv\" (UniqueName: \"kubernetes.io/projected/93e17f1e-700c-49a0-b90f-fc8d8571801b-kube-api-access-jh7zv\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:24.660810 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.660801 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-trusted-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:24.660810 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.660814 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-serving-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:24.661060 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.660825 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93e17f1e-700c-49a0-b90f-fc8d8571801b-console-oauth-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:24.661060 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:24.660838 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93e17f1e-700c-49a0-b90f-fc8d8571801b-oauth-serving-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:25.500611 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:25.500583 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7cf449b5-68bxz" Jul 21 14:34:25.518718 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:25.518688 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b7cf449b5-68bxz"] Jul 21 14:34:25.523779 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:25.523754 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b7cf449b5-68bxz"] Jul 21 14:34:26.915036 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:26.914999 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e17f1e-700c-49a0-b90f-fc8d8571801b" path="/var/lib/kubelet/pods/93e17f1e-700c-49a0-b90f-fc8d8571801b/volumes" Jul 21 14:34:30.076643 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:30.076612 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:34:30.080467 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:30.080442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66b5c874f-8fb4d" Jul 21 14:34:35.665056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:35.664999 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6695478675-mckkj" podUID="9cb85d09-9d20-4aa3-8817-2f795fd49484" containerName="console" containerID="cri-o://cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067" gracePeriod=15 Jul 21 14:34:35.908511 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:35.908484 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:34:36.063021 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.062997 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-trusted-ca-bundle\") pod \"9cb85d09-9d20-4aa3-8817-2f795fd49484\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " Jul 21 14:34:36.063185 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063031 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gn92\" (UniqueName: \"kubernetes.io/projected/9cb85d09-9d20-4aa3-8817-2f795fd49484-kube-api-access-2gn92\") pod \"9cb85d09-9d20-4aa3-8817-2f795fd49484\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " Jul 21 14:34:36.063185 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-oauth-serving-cert\") pod \"9cb85d09-9d20-4aa3-8817-2f795fd49484\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " Jul 21 14:34:36.063185 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063083 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-config\") pod \"9cb85d09-9d20-4aa3-8817-2f795fd49484\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " Jul 21 14:34:36.063185 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-serving-cert\") pod \"9cb85d09-9d20-4aa3-8817-2f795fd49484\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " Jul 21 14:34:36.063434 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063238 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-oauth-config\") pod \"9cb85d09-9d20-4aa3-8817-2f795fd49484\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " Jul 21 14:34:36.063434 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063300 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-service-ca\") pod \"9cb85d09-9d20-4aa3-8817-2f795fd49484\" (UID: \"9cb85d09-9d20-4aa3-8817-2f795fd49484\") " Jul 21 14:34:36.063534 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063479 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9cb85d09-9d20-4aa3-8817-2f795fd49484" (UID: "9cb85d09-9d20-4aa3-8817-2f795fd49484"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:36.063534 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063514 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9cb85d09-9d20-4aa3-8817-2f795fd49484" (UID: "9cb85d09-9d20-4aa3-8817-2f795fd49484"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:36.063534 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063521 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-config" (OuterVolumeSpecName: "console-config") pod "9cb85d09-9d20-4aa3-8817-2f795fd49484" (UID: "9cb85d09-9d20-4aa3-8817-2f795fd49484"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:36.063714 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063614 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-trusted-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:36.063831 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063722 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-oauth-serving-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:36.063831 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063736 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:36.063831 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.063774 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-service-ca" (OuterVolumeSpecName: "service-ca") pod "9cb85d09-9d20-4aa3-8817-2f795fd49484" (UID: "9cb85d09-9d20-4aa3-8817-2f795fd49484"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:34:36.065432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.065405 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9cb85d09-9d20-4aa3-8817-2f795fd49484" (UID: "9cb85d09-9d20-4aa3-8817-2f795fd49484"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:34:36.065809 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.065788 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9cb85d09-9d20-4aa3-8817-2f795fd49484" (UID: "9cb85d09-9d20-4aa3-8817-2f795fd49484"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:34:36.065885 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.065819 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb85d09-9d20-4aa3-8817-2f795fd49484-kube-api-access-2gn92" (OuterVolumeSpecName: "kube-api-access-2gn92") pod "9cb85d09-9d20-4aa3-8817-2f795fd49484" (UID: "9cb85d09-9d20-4aa3-8817-2f795fd49484"). InnerVolumeSpecName "kube-api-access-2gn92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:34:36.165064 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.165035 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb85d09-9d20-4aa3-8817-2f795fd49484-service-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:36.165064 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.165062 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gn92\" (UniqueName: \"kubernetes.io/projected/9cb85d09-9d20-4aa3-8817-2f795fd49484-kube-api-access-2gn92\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:36.165064 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.165073 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-serving-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:36.165300 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.165082 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cb85d09-9d20-4aa3-8817-2f795fd49484-console-oauth-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:34:36.548938 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.548898 2559 generic.go:356] "Generic (PLEG): container finished" podID="9cb85d09-9d20-4aa3-8817-2f795fd49484" containerID="cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067" exitCode=0 Jul 21 14:34:36.549101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.548952 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6695478675-mckkj" Jul 21 14:34:36.549101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.548970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6695478675-mckkj" event={"ID":"9cb85d09-9d20-4aa3-8817-2f795fd49484","Type":"ContainerDied","Data":"cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067"} Jul 21 14:34:36.549101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.549013 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6695478675-mckkj" event={"ID":"9cb85d09-9d20-4aa3-8817-2f795fd49484","Type":"ContainerDied","Data":"0106744122ba60843420ff6b72d8103f5e707897de32e78b7bb10200b867813e"} Jul 21 14:34:36.549101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.549031 2559 scope.go:117] "RemoveContainer" containerID="cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067" Jul 21 14:34:36.557714 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.557697 2559 scope.go:117] "RemoveContainer" containerID="cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067" Jul 21 14:34:36.557978 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:34:36.557960 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067\": container with ID starting with cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067 not found: ID does not exist" containerID="cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067" Jul 21 14:34:36.558025 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.557987 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067"} err="failed to get container status \"cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067\": rpc error: code = NotFound desc = could not find container \"cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067\": container with ID starting with cf85c078d02d327134bf7ca8ad5e64ed009f443b116510c524fb65166d0d2067 not found: ID does not exist" Jul 21 14:34:36.569512 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.569489 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6695478675-mckkj"] Jul 21 14:34:36.573058 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.573035 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6695478675-mckkj"] Jul 21 14:34:36.916874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:36.916797 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cb85d09-9d20-4aa3-8817-2f795fd49484" path="/var/lib/kubelet/pods/9cb85d09-9d20-4aa3-8817-2f795fd49484/volumes" Jul 21 14:34:51.922487 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:51.922432 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:34:51.942435 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:51.942408 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:34:52.627947 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:34:52.627917 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:01.509593 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:01.509555 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:01.609634 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:01.609603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:04.121195 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:04.121161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:04.187055 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:04.187015 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:04.645993 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:04.645949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:04.747138 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:04.747098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:05.789038 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:05.789001 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:35:05.789473 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:05.789325 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="alertmanager" containerID="cri-o://592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09" gracePeriod=120 Jul 21 14:35:05.789473 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:05.789363 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy" containerID="cri-o://a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d" gracePeriod=120 Jul 21 14:35:05.789473 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:05.789397 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="prom-label-proxy" containerID="cri-o://bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08" gracePeriod=120 Jul 21 14:35:05.789473 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:05.789425 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-web" containerID="cri-o://a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89" gracePeriod=120 Jul 21 14:35:05.789473 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:05.789448 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="config-reloader" containerID="cri-o://8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf" gracePeriod=120 Jul 21 14:35:05.789766 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:05.789424 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-metric" containerID="cri-o://4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c" gracePeriod=120 Jul 21 14:35:06.674667 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674629 2559 generic.go:356] "Generic (PLEG): container finished" podID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerID="bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08" exitCode=0 Jul 21 14:35:06.674667 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674658 2559 generic.go:356] "Generic (PLEG): container finished" podID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerID="4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c" exitCode=0 Jul 21 14:35:06.674667 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674666 2559 generic.go:356] "Generic (PLEG): container finished" podID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerID="a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d" exitCode=0 Jul 21 14:35:06.674667 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674671 2559 generic.go:356] "Generic (PLEG): container finished" podID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerID="8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf" exitCode=0 Jul 21 14:35:06.674667 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674677 2559 generic.go:356] "Generic (PLEG): container finished" podID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerID="592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09" exitCode=0 Jul 21 14:35:06.675076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674702 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08"} Jul 21 14:35:06.675076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674876 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c"} Jul 21 14:35:06.675076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d"} Jul 21 14:35:06.675076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.674970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf"} Jul 21 14:35:06.675076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:06.675010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09"} Jul 21 14:35:07.033917 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.033894 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.134535 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134501 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-out\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134548 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134568 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kntv\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-kube-api-access-5kntv\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134609 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-cluster-tls-config\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134637 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-metrics-client-ca\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134910 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134699 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-web-config\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134910 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134723 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-main-db\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134910 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-web\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134910 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134790 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134910 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134817 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-main-tls\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134910 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134865 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-tls-assets\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.134910 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134889 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-volume\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.135334 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.134928 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle\") pod \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\" (UID: \"eeded95d-e423-4ab5-b9a4-0e770f2b787b\") " Jul 21 14:35:07.135584 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.135483 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:07.135584 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.135548 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:07.137261 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.137210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:35:07.139121 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139076 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-kube-api-access-5kntv" (OuterVolumeSpecName: "kube-api-access-5kntv") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "kube-api-access-5kntv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:35:07.139121 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139096 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-out" (OuterVolumeSpecName: "config-out") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:35:07.139121 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139093 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:07.139369 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139295 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:35:07.139369 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139333 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:07.139482 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139469 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:07.139668 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139646 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-volume" (OuterVolumeSpecName: "config-volume") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:07.139966 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.139938 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:07.143299 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.143175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:07.150406 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.150375 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-web-config" (OuterVolumeSpecName: "web-config") pod "eeded95d-e423-4ab5-b9a4-0e770f2b787b" (UID: "eeded95d-e423-4ab5-b9a4-0e770f2b787b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:07.236103 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236049 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236103 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236071 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kntv\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-kube-api-access-5kntv\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236103 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236082 2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-cluster-tls-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236103 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236090 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-metrics-client-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236122 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-web-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236132 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-main-db\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236140 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236150 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236160 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-secret-alertmanager-main-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236168 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeded95d-e423-4ab5-b9a4-0e770f2b787b-tls-assets\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236176 2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-volume\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236187 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeded95d-e423-4ab5-b9a4-0e770f2b787b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.236302 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.236198 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeded95d-e423-4ab5-b9a4-0e770f2b787b-config-out\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:07.682828 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.682797 2559 generic.go:356] "Generic (PLEG): container finished" podID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerID="a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89" exitCode=0 Jul 21 14:35:07.683023 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.682859 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89"} Jul 21 14:35:07.683023 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.682882 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.683023 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.682948 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeded95d-e423-4ab5-b9a4-0e770f2b787b","Type":"ContainerDied","Data":"5a9995e4c7d3ea18d42b7ceaccf58c375948423e3c06f5ef8ae176bc894e0847"} Jul 21 14:35:07.683023 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.682983 2559 scope.go:117] "RemoveContainer" containerID="bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08" Jul 21 14:35:07.690723 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.690701 2559 scope.go:117] "RemoveContainer" containerID="4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c" Jul 21 14:35:07.697399 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.697381 2559 scope.go:117] "RemoveContainer" containerID="a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d" Jul 21 14:35:07.704072 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.704035 2559 scope.go:117] "RemoveContainer" containerID="a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89" Jul 21 14:35:07.706769 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.706747 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:35:07.710840 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.710818 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:35:07.712062 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.712050 2559 scope.go:117] "RemoveContainer" containerID="8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf" Jul 21 14:35:07.718825 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.718810 2559 scope.go:117] "RemoveContainer" containerID="592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09" Jul 21 14:35:07.725564 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.725535 2559 scope.go:117] "RemoveContainer" containerID="c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1" Jul 21 14:35:07.732691 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.732675 2559 scope.go:117] "RemoveContainer" containerID="bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08" Jul 21 14:35:07.732970 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:07.732952 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08\": container with ID starting with bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08 not found: ID does not exist" containerID="bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08" Jul 21 14:35:07.733019 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.732980 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08"} err="failed to get container status \"bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08\": rpc error: code = NotFound desc = could not find container \"bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08\": container with ID starting with bfe145b7dd20727e3599c6b9cec2ed9c7916f54739bb235de1b30006a6da8a08 not found: ID does not exist" Jul 21 14:35:07.733019 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.733001 2559 scope.go:117] "RemoveContainer" containerID="4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c" Jul 21 14:35:07.733258 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:07.733209 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c\": container with ID starting with 4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c not found: ID does not exist" containerID="4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c" Jul 21 14:35:07.733345 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.733268 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c"} err="failed to get container status \"4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c\": rpc error: code = NotFound desc = could not find container \"4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c\": container with ID starting with 4e4fd73b0e024cf8b65a2e6eb33d7757d03716b7dc87aa91cd4a9e6722fc187c not found: ID does not exist" Jul 21 14:35:07.733345 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.733291 2559 scope.go:117] "RemoveContainer" containerID="a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d" Jul 21 14:35:07.733563 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:07.733546 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d\": container with ID starting with a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d not found: ID does not exist" containerID="a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d" Jul 21 14:35:07.733615 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.733567 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d"} err="failed to get container status \"a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d\": rpc error: code = NotFound desc = could not find container \"a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d\": container with ID starting with a8aba15c21f74de7e8da6d99464d01002500c0a46699b61d94b0be7eceb94e5d not found: ID does not exist" Jul 21 14:35:07.733615 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.733581 2559 scope.go:117] "RemoveContainer" containerID="a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89" Jul 21 14:35:07.733771 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:07.733756 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89\": container with ID starting with a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89 not found: ID does not exist" containerID="a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89" Jul 21 14:35:07.733808 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.733774 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89"} err="failed to get container status \"a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89\": rpc error: code = NotFound desc = could not find container \"a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89\": container with ID starting with a68b4e131a0e63d87cb07c972aeb539d6203fe4df3718f2a632422ce51e68e89 not found: ID does not exist" Jul 21 14:35:07.733808 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.733787 2559 scope.go:117] "RemoveContainer" containerID="8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf" Jul 21 14:35:07.734015 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:07.733996 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf\": container with ID starting with 8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf not found: ID does not exist" containerID="8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf" Jul 21 14:35:07.734059 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.734019 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf"} err="failed to get container status \"8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf\": rpc error: code = NotFound desc = could not find container \"8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf\": container with ID starting with 8ef633044b38f7fe90bbaae37e9bc5ea6d77e73237cbd55d2bd8aa5a33988ccf not found: ID does not exist" Jul 21 14:35:07.734059 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.734033 2559 scope.go:117] "RemoveContainer" containerID="592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09" Jul 21 14:35:07.734332 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:07.734315 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09\": container with ID starting with 592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09 not found: ID does not exist" containerID="592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09" Jul 21 14:35:07.734378 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.734336 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09"} err="failed to get container status \"592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09\": rpc error: code = NotFound desc = could not find container \"592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09\": container with ID starting with 592a8aaed2abdbaa36bc7d8837990a87f5ea0752716200d972f9ab830f4cea09 not found: ID does not exist" Jul 21 14:35:07.734378 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.734350 2559 scope.go:117] "RemoveContainer" containerID="c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1" Jul 21 14:35:07.734565 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:07.734547 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1\": container with ID starting with c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1 not found: ID does not exist" containerID="c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1" Jul 21 14:35:07.734636 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.734569 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1"} err="failed to get container status \"c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1\": rpc error: code = NotFound desc = could not find container \"c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1\": container with ID starting with c6fa9b95443617001ddda11aa1859097e9217abadf3e2a57603c6c002d95bbb1 not found: ID does not exist" Jul 21 14:35:07.737445 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737425 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:35:07.737746 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737733 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy" Jul 21 14:35:07.737789 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737748 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy" Jul 21 14:35:07.737789 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737761 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-web" Jul 21 14:35:07.737789 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737768 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-web" Jul 21 14:35:07.737789 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737777 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cb85d09-9d20-4aa3-8817-2f795fd49484" containerName="console" Jul 21 14:35:07.737789 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737783 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cb85d09-9d20-4aa3-8817-2f795fd49484" containerName="console" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737793 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0631a7-8761-4ae0-9bc6-4b7813af7f71" containerName="console" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737801 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0631a7-8761-4ae0-9bc6-4b7813af7f71" containerName="console" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737813 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="alertmanager" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737818 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="alertmanager" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737825 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="config-reloader" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737830 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="config-reloader" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737837 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-metric" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737841 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-metric" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737848 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="prom-label-proxy" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737853 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="prom-label-proxy" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737859 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="init-config-reloader" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737864 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="init-config-reloader" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737876 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e17f1e-700c-49a0-b90f-fc8d8571801b" containerName="console" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737883 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e17f1e-700c-49a0-b90f-fc8d8571801b" containerName="console" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737926 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="alertmanager" Jul 21 14:35:07.737933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737934 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="prom-label-proxy" Jul 21 14:35:07.738432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737943 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-web" Jul 21 14:35:07.738432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737954 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy-metric" Jul 21 14:35:07.738432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737961 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cb85d09-9d20-4aa3-8817-2f795fd49484" containerName="console" Jul 21 14:35:07.738432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737967 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="config-reloader" Jul 21 14:35:07.738432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737973 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e17f1e-700c-49a0-b90f-fc8d8571801b" containerName="console" Jul 21 14:35:07.738432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737977 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" containerName="kube-rbac-proxy" Jul 21 14:35:07.738432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.737982 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0631a7-8761-4ae0-9bc6-4b7813af7f71" containerName="console" Jul 21 14:35:07.741502 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.741484 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.744184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744055 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 21 14:35:07.744184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744067 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 21 14:35:07.744184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744099 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 21 14:35:07.744184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744067 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 21 14:35:07.744184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744146 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z5vlw\"" Jul 21 14:35:07.744184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744072 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 21 14:35:07.744733 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744379 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 21 14:35:07.744733 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744396 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 21 14:35:07.744733 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.744513 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 21 14:35:07.749683 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.749666 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 21 14:35:07.756710 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.756690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:35:07.840436 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840408 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840436 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-web-config\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840613 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840613 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840613 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840506 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840613 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840613 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-config-out\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6q8n\" (UniqueName: \"kubernetes.io/projected/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-kube-api-access-k6q8n\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840666 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840797 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.840997 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.840829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.941861 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.941776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.941861 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.941830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942044 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.941988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942044 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942044 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-web-config\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942205 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942205 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942205 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-config-out\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942398 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6q8n\" (UniqueName: \"kubernetes.io/projected/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-kube-api-access-k6q8n\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.942593 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.943790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.942783 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.943790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.943276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.944790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.944770 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.944885 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.944812 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-web-config\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.945021 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.944995 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.945120 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.945100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.945184 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.945100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.945673 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.945645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.945770 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.945710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-config-out\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.945770 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.945762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.946012 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.945997 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.946919 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.946903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:07.954845 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:07.954825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6q8n\" (UniqueName: \"kubernetes.io/projected/de6bf28c-c86b-4b74-8b8c-ba96fda955b2-kube-api-access-k6q8n\") pod \"alertmanager-main-0\" (UID: \"de6bf28c-c86b-4b74-8b8c-ba96fda955b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:08.050855 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:08.050821 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 14:35:08.186781 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:08.186755 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 14:35:08.189233 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:35:08.189189 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6bf28c_c86b_4b74_8b8c_ba96fda955b2.slice/crio-8df121068c737ccd206eb3dea68c9f6f55ab9d78a696793bdd2b794051b05d59 WatchSource:0}: Error finding container 8df121068c737ccd206eb3dea68c9f6f55ab9d78a696793bdd2b794051b05d59: Status 404 returned error can't find the container with id 8df121068c737ccd206eb3dea68c9f6f55ab9d78a696793bdd2b794051b05d59 Jul 21 14:35:08.689098 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:08.689063 2559 generic.go:356] "Generic (PLEG): container finished" podID="de6bf28c-c86b-4b74-8b8c-ba96fda955b2" containerID="455c848fe59e1418e9092316d46240dc5e68094c8d251551a21d34d83ce42a0c" exitCode=0 Jul 21 14:35:08.689288 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:08.689150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerDied","Data":"455c848fe59e1418e9092316d46240dc5e68094c8d251551a21d34d83ce42a0c"} Jul 21 14:35:08.689288 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:08.689237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerStarted","Data":"8df121068c737ccd206eb3dea68c9f6f55ab9d78a696793bdd2b794051b05d59"} Jul 21 14:35:08.915017 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:08.914992 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeded95d-e423-4ab5-b9a4-0e770f2b787b" path="/var/lib/kubelet/pods/eeded95d-e423-4ab5-b9a4-0e770f2b787b/volumes" Jul 21 14:35:09.697703 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:09.697673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerStarted","Data":"1439f85b04ea5618ac071c393b9cd45e28d31be361dddf1e9e288fcdb7fa3c50"} Jul 21 14:35:09.698169 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:09.697750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerStarted","Data":"eb6f8be174675ae746270b24042375e8d6811cbbed85469fcdea62c5f6576af8"} Jul 21 14:35:09.698169 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:09.697798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerStarted","Data":"035b6378f9a4b71ac77f3be0d5899fddeb630fa1628a709877f95ba16d289845"} Jul 21 14:35:09.698169 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:09.697838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerStarted","Data":"d297765c2867d7bc58b122fba223e1956ae597994d5d489863ec21e4e1474ee6"} Jul 21 14:35:09.698169 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:09.697885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerStarted","Data":"34c07e5d11fc417148b5c045c60a7ce790f6edfd53016fa4d6461cc70f010d0c"} Jul 21 14:35:09.698169 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:09.697936 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de6bf28c-c86b-4b74-8b8c-ba96fda955b2","Type":"ContainerStarted","Data":"3e8270425d7152d558fd2f97667e30fd8a0876dee8ddfbac87edc72d86e19cbe"} Jul 21 14:35:09.725693 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:09.725630 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.708048183 podStartE2EDuration="2.725613114s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.564931ms" isStatefulPod=false podCreationTimestamp="2026-07-21 14:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:35:09.723127485 +0000 UTC m=+154.211318690" watchObservedRunningTime="2026-07-21 14:35:09.725613114 +0000 UTC m=+154.213804317" Jul 21 14:35:10.045318 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.045279 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:35:10.045939 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.045879 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="prometheus" containerID="cri-o://6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" gracePeriod=600 Jul 21 14:35:10.046635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.046101 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy" containerID="cri-o://5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" gracePeriod=600 Jul 21 14:35:10.046635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.046144 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" gracePeriod=600 Jul 21 14:35:10.046635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.046163 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="thanos-sidecar" containerID="cri-o://9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" gracePeriod=600 Jul 21 14:35:10.046635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.046181 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-web" containerID="cri-o://ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" gracePeriod=600 Jul 21 14:35:10.046635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.046197 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="config-reloader" containerID="cri-o://fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" gracePeriod=600 Jul 21 14:35:10.296337 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.296270 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.369460 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369430 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-metrics-client-certs\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369460 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369469 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-metrics-client-ca\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369526 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-serving-certs-ca-bundle\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369579 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-web-config\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369808 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369708 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369808 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369765 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-kubelet-serving-ca-bundle\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369808 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369793 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-db\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369828 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-1\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369856 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-trusted-ca-bundle\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369880 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-0\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369920 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:10.369955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369933 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-config-out\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.369955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369955 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-2\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369968 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.369980 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-grpc-tls\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.370033 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-tls-assets\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.370108 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-tls\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.370137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-thanos-prometheus-http-client-file\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.370169 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-config\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.370194 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-kube-rbac-proxy\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.370360 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.370243 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd2mr\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-kube-api-access-pd2mr\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.371111 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.371085 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:35:10.372065 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.371298 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"18441986-b4c9-40ed-9090-51080bca018f\" (UID: \"18441986-b4c9-40ed-9090-51080bca018f\") " Jul 21 14:35:10.372065 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.371614 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.372065 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.371635 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-db\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.372065 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.371651 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-metrics-client-ca\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.372065 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.371682 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:10.372384 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.372101 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:10.372384 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.372250 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:10.372934 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.372530 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:10.373573 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.372884 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:35:10.377156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.377030 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-kube-api-access-pd2mr" (OuterVolumeSpecName: "kube-api-access-pd2mr") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "kube-api-access-pd2mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:35:10.377705 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.377669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-config-out" (OuterVolumeSpecName: "config-out") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:35:10.377705 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.377695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.377871 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.377817 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.377871 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.377842 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-config" (OuterVolumeSpecName: "config") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.378000 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.377976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:35:10.378162 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.378089 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.378277 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.378240 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.378599 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.378566 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.378882 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.378863 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.379136 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.379116 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.388014 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.387991 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-web-config" (OuterVolumeSpecName: "web-config") pod "18441986-b4c9-40ed-9090-51080bca018f" (UID: "18441986-b4c9-40ed-9090-51080bca018f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:35:10.472826 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472788 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18441986-b4c9-40ed-9090-51080bca018f-config-out\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.472826 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472820 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.472826 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472831 2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-grpc-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472840 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-tls-assets\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472849 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472859 2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472869 2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472877 2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-kube-rbac-proxy\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472886 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2mr\" (UniqueName: \"kubernetes.io/projected/18441986-b4c9-40ed-9090-51080bca018f-kube-api-access-pd2mr\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472895 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472904 2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-metrics-client-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472912 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-web-config\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472921 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18441986-b4c9-40ed-9090-51080bca018f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472929 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472938 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472945 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.473056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.472953 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18441986-b4c9-40ed-9090-51080bca018f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704673 2559 generic.go:356] "Generic (PLEG): container finished" podID="18441986-b4c9-40ed-9090-51080bca018f" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" exitCode=0 Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704700 2559 generic.go:356] "Generic (PLEG): container finished" podID="18441986-b4c9-40ed-9090-51080bca018f" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" exitCode=0 Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704708 2559 generic.go:356] "Generic (PLEG): container finished" podID="18441986-b4c9-40ed-9090-51080bca018f" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" exitCode=0 Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704722 2559 generic.go:356] "Generic (PLEG): container finished" podID="18441986-b4c9-40ed-9090-51080bca018f" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" exitCode=0 Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704730 2559 generic.go:356] "Generic (PLEG): container finished" podID="18441986-b4c9-40ed-9090-51080bca018f" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" exitCode=0 Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704737 2559 generic.go:356] "Generic (PLEG): container finished" podID="18441986-b4c9-40ed-9090-51080bca018f" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" exitCode=0 Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} Jul 21 14:35:10.704783 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704780 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.705551 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} Jul 21 14:35:10.705551 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704886 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} Jul 21 14:35:10.705551 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704910 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} Jul 21 14:35:10.705551 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} Jul 21 14:35:10.705551 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704927 2559 scope.go:117] "RemoveContainer" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.705551 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} Jul 21 14:35:10.705551 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.704984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18441986-b4c9-40ed-9090-51080bca018f","Type":"ContainerDied","Data":"e2e9228c0138b876c8932a3d47b57525ac054191b9c60dc201eb385a4b9c88f8"} Jul 21 14:35:10.713608 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.713590 2559 scope.go:117] "RemoveContainer" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.720530 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.720513 2559 scope.go:117] "RemoveContainer" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.728542 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.728520 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:35:10.729097 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.729082 2559 scope.go:117] "RemoveContainer" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.732662 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.732643 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:35:10.736124 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.736110 2559 scope.go:117] "RemoveContainer" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.742715 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.742699 2559 scope.go:117] "RemoveContainer" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.749941 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.749923 2559 scope.go:117] "RemoveContainer" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.756743 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.756718 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:35:10.757083 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757066 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="config-reloader" Jul 21 14:35:10.757156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757086 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="config-reloader" Jul 21 14:35:10.757156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757099 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="thanos-sidecar" Jul 21 14:35:10.757156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757107 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="thanos-sidecar" Jul 21 14:35:10.757156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757133 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-thanos" Jul 21 14:35:10.757156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757143 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-thanos" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757160 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="init-config-reloader" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757169 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="init-config-reloader" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757179 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757188 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757198 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-web" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757207 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-web" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757244 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="prometheus" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757253 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="prometheus" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757335 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="prometheus" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757350 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="thanos-sidecar" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757360 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757372 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-thanos" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757382 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="kube-rbac-proxy-web" Jul 21 14:35:10.757417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757392 2559 scope.go:117] "RemoveContainer" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.757897 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757393 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18441986-b4c9-40ed-9090-51080bca018f" containerName="config-reloader" Jul 21 14:35:10.757897 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:10.757671 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": container with ID starting with 84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74 not found: ID does not exist" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.757897 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757697 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} err="failed to get container status \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": rpc error: code = NotFound desc = could not find container \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": container with ID starting with 84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74 not found: ID does not exist" Jul 21 14:35:10.757897 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757718 2559 scope.go:117] "RemoveContainer" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.758035 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:10.757957 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": container with ID starting with 5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a not found: ID does not exist" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.758035 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757978 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} err="failed to get container status \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": rpc error: code = NotFound desc = could not find container \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": container with ID starting with 5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a not found: ID does not exist" Jul 21 14:35:10.758035 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.757997 2559 scope.go:117] "RemoveContainer" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.758253 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:10.758212 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": container with ID starting with ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451 not found: ID does not exist" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.758313 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.758264 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} err="failed to get container status \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": rpc error: code = NotFound desc = could not find container \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": container with ID starting with ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451 not found: ID does not exist" Jul 21 14:35:10.758313 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.758286 2559 scope.go:117] "RemoveContainer" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.758553 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:10.758530 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": container with ID starting with 9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0 not found: ID does not exist" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.758642 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.758556 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} err="failed to get container status \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": rpc error: code = NotFound desc = could not find container \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": container with ID starting with 9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0 not found: ID does not exist" Jul 21 14:35:10.758642 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.758569 2559 scope.go:117] "RemoveContainer" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.758825 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:10.758801 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": container with ID starting with fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c not found: ID does not exist" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.758867 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.758835 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} err="failed to get container status \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": rpc error: code = NotFound desc = could not find container \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": container with ID starting with fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c not found: ID does not exist" Jul 21 14:35:10.758867 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.758855 2559 scope.go:117] "RemoveContainer" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.759185 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:10.759163 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": container with ID starting with 6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854 not found: ID does not exist" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.759303 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.759190 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} err="failed to get container status \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": rpc error: code = NotFound desc = could not find container \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": container with ID starting with 6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854 not found: ID does not exist" Jul 21 14:35:10.759303 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.759211 2559 scope.go:117] "RemoveContainer" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.759569 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:35:10.759552 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": container with ID starting with 89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43 not found: ID does not exist" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.759633 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.759579 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43"} err="failed to get container status \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": rpc error: code = NotFound desc = could not find container \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": container with ID starting with 89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43 not found: ID does not exist" Jul 21 14:35:10.759633 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.759600 2559 scope.go:117] "RemoveContainer" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.759855 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.759839 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} err="failed to get container status \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": rpc error: code = NotFound desc = could not find container \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": container with ID starting with 84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74 not found: ID does not exist" Jul 21 14:35:10.759927 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.759858 2559 scope.go:117] "RemoveContainer" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.760084 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760066 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} err="failed to get container status \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": rpc error: code = NotFound desc = could not find container \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": container with ID starting with 5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a not found: ID does not exist" Jul 21 14:35:10.760154 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760086 2559 scope.go:117] "RemoveContainer" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.760328 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760309 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} err="failed to get container status \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": rpc error: code = NotFound desc = could not find container \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": container with ID starting with ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451 not found: ID does not exist" Jul 21 14:35:10.760371 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760331 2559 scope.go:117] "RemoveContainer" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.760572 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760554 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} err="failed to get container status \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": rpc error: code = NotFound desc = could not find container \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": container with ID starting with 9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0 not found: ID does not exist" Jul 21 14:35:10.760626 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760573 2559 scope.go:117] "RemoveContainer" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.760878 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760857 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} err="failed to get container status \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": rpc error: code = NotFound desc = could not find container \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": container with ID starting with fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c not found: ID does not exist" Jul 21 14:35:10.760962 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.760878 2559 scope.go:117] "RemoveContainer" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.761167 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761142 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} err="failed to get container status \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": rpc error: code = NotFound desc = could not find container \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": container with ID starting with 6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854 not found: ID does not exist" Jul 21 14:35:10.761167 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761161 2559 scope.go:117] "RemoveContainer" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.761434 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761404 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43"} err="failed to get container status \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": rpc error: code = NotFound desc = could not find container \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": container with ID starting with 89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43 not found: ID does not exist" Jul 21 14:35:10.761434 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761424 2559 scope.go:117] "RemoveContainer" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.761675 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761656 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} err="failed to get container status \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": rpc error: code = NotFound desc = could not find container \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": container with ID starting with 84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74 not found: ID does not exist" Jul 21 14:35:10.761723 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761675 2559 scope.go:117] "RemoveContainer" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.761906 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761884 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} err="failed to get container status \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": rpc error: code = NotFound desc = could not find container \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": container with ID starting with 5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a not found: ID does not exist" Jul 21 14:35:10.761906 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.761905 2559 scope.go:117] "RemoveContainer" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.762096 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762057 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.762146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762123 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} err="failed to get container status \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": rpc error: code = NotFound desc = could not find container \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": container with ID starting with ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451 not found: ID does not exist" Jul 21 14:35:10.762146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762138 2559 scope.go:117] "RemoveContainer" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.762499 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762403 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} err="failed to get container status \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": rpc error: code = NotFound desc = could not find container \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": container with ID starting with 9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0 not found: ID does not exist" Jul 21 14:35:10.762499 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762432 2559 scope.go:117] "RemoveContainer" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.762710 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762682 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} err="failed to get container status \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": rpc error: code = NotFound desc = could not find container \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": container with ID starting with fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c not found: ID does not exist" Jul 21 14:35:10.762757 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762713 2559 scope.go:117] "RemoveContainer" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.762960 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762942 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} err="failed to get container status \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": rpc error: code = NotFound desc = could not find container \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": container with ID starting with 6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854 not found: ID does not exist" Jul 21 14:35:10.763008 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.762961 2559 scope.go:117] "RemoveContainer" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.763290 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.763265 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43"} err="failed to get container status \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": rpc error: code = NotFound desc = could not find container \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": container with ID starting with 89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43 not found: ID does not exist" Jul 21 14:35:10.763343 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.763291 2559 scope.go:117] "RemoveContainer" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.763562 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.763544 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} err="failed to get container status \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": rpc error: code = NotFound desc = could not find container \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": container with ID starting with 84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74 not found: ID does not exist" Jul 21 14:35:10.763608 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.763563 2559 scope.go:117] "RemoveContainer" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.763839 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.763817 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} err="failed to get container status \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": rpc error: code = NotFound desc = could not find container \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": container with ID starting with 5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a not found: ID does not exist" Jul 21 14:35:10.763926 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.763841 2559 scope.go:117] "RemoveContainer" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.764094 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764075 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} err="failed to get container status \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": rpc error: code = NotFound desc = could not find container \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": container with ID starting with ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451 not found: ID does not exist" Jul 21 14:35:10.764136 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764095 2559 scope.go:117] "RemoveContainer" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.764335 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764315 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} err="failed to get container status \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": rpc error: code = NotFound desc = could not find container \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": container with ID starting with 9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0 not found: ID does not exist" Jul 21 14:35:10.764335 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764335 2559 scope.go:117] "RemoveContainer" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.764585 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764567 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} err="failed to get container status \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": rpc error: code = NotFound desc = could not find container \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": container with ID starting with fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c not found: ID does not exist" Jul 21 14:35:10.764628 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764585 2559 scope.go:117] "RemoveContainer" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.764761 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764743 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 14:35:10.764907 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764875 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 14:35:10.764983 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764917 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} err="failed to get container status \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": rpc error: code = NotFound desc = could not find container \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": container with ID starting with 6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854 not found: ID does not exist" Jul 21 14:35:10.764983 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764945 2559 scope.go:117] "RemoveContainer" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.764983 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.764977 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 14:35:10.765141 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765089 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 14:35:10.765339 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 14:35:10.765481 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765355 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 14:35:10.765543 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765472 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43"} err="failed to get container status \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": rpc error: code = NotFound desc = could not find container \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": container with ID starting with 89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43 not found: ID does not exist" Jul 21 14:35:10.765543 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765494 2559 scope.go:117] "RemoveContainer" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.765798 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765689 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 14:35:10.765798 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765709 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 14:35:10.765798 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765721 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} err="failed to get container status \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": rpc error: code = NotFound desc = could not find container \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": container with ID starting with 84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74 not found: ID does not exist" Jul 21 14:35:10.765798 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765741 2559 scope.go:117] "RemoveContainer" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.766006 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.765941 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fbr435ta36n1k\"" Jul 21 14:35:10.766006 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.766002 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 14:35:10.767948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.767917 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} err="failed to get container status \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": rpc error: code = NotFound desc = could not find container \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": container with ID starting with 5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a not found: ID does not exist" Jul 21 14:35:10.767948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.767944 2559 scope.go:117] "RemoveContainer" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.767948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.767967 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 14:35:10.767948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768044 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h5k9q\"" Jul 21 14:35:10.767948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768065 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 14:35:10.768571 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768325 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} err="failed to get container status \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": rpc error: code = NotFound desc = could not find container \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": container with ID starting with ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451 not found: ID does not exist" Jul 21 14:35:10.768571 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768349 2559 scope.go:117] "RemoveContainer" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.768571 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768440 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 14:35:10.768725 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768615 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} err="failed to get container status \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": rpc error: code = NotFound desc = could not find container \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": container with ID starting with 9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0 not found: ID does not exist" Jul 21 14:35:10.768725 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768637 2559 scope.go:117] "RemoveContainer" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.768971 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768944 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} err="failed to get container status \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": rpc error: code = NotFound desc = could not find container \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": container with ID starting with fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c not found: ID does not exist" Jul 21 14:35:10.768971 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.768970 2559 scope.go:117] "RemoveContainer" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.769355 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.769329 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} err="failed to get container status \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": rpc error: code = NotFound desc = could not find container \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": container with ID starting with 6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854 not found: ID does not exist" Jul 21 14:35:10.769460 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.769358 2559 scope.go:117] "RemoveContainer" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.769621 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.769600 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43"} err="failed to get container status \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": rpc error: code = NotFound desc = could not find container \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": container with ID starting with 89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43 not found: ID does not exist" Jul 21 14:35:10.769621 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.769619 2559 scope.go:117] "RemoveContainer" containerID="84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74" Jul 21 14:35:10.769902 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.769851 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74"} err="failed to get container status \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": rpc error: code = NotFound desc = could not find container \"84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74\": container with ID starting with 84f3a8d37b8903d42792e2269015f94872ab8d8aa9543f0df1ec464dfcbf2d74 not found: ID does not exist" Jul 21 14:35:10.769902 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.769879 2559 scope.go:117] "RemoveContainer" containerID="5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a" Jul 21 14:35:10.770175 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.770148 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a"} err="failed to get container status \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": rpc error: code = NotFound desc = could not find container \"5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a\": container with ID starting with 5f76e000f9d16ecaba83ed885066e486480bb2e1718078eaff9aebd25b26592a not found: ID does not exist" Jul 21 14:35:10.770175 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.770173 2559 scope.go:117] "RemoveContainer" containerID="ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451" Jul 21 14:35:10.770357 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.770333 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 14:35:10.770577 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.770549 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451"} err="failed to get container status \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": rpc error: code = NotFound desc = could not find container \"ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451\": container with ID starting with ec9711f6d0a386668314143167f2ec85d7a18d0ce07657e233d5d6260bcaf451 not found: ID does not exist" Jul 21 14:35:10.770656 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.770579 2559 scope.go:117] "RemoveContainer" containerID="9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0" Jul 21 14:35:10.770831 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.770804 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0"} err="failed to get container status \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": rpc error: code = NotFound desc = could not find container \"9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0\": container with ID starting with 9a7e4a945630d90df156041b9bca96f71e2a61a7bc4e45e6cc572074812ddcb0 not found: ID does not exist" Jul 21 14:35:10.770890 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.770833 2559 scope.go:117] "RemoveContainer" containerID="fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c" Jul 21 14:35:10.771136 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.771114 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c"} err="failed to get container status \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": rpc error: code = NotFound desc = could not find container \"fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c\": container with ID starting with fa1e59fed35097487c69945e8d793dd2199a2bc8cc3f3a8972781795dbd5a89c not found: ID does not exist" Jul 21 14:35:10.771211 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.771137 2559 scope.go:117] "RemoveContainer" containerID="6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854" Jul 21 14:35:10.771452 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.771427 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854"} err="failed to get container status \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": rpc error: code = NotFound desc = could not find container \"6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854\": container with ID starting with 6814b9181beced7af4a01a619108d79aaaed0a7bb77d07da20bd472e45d88854 not found: ID does not exist" Jul 21 14:35:10.771530 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.771453 2559 scope.go:117] "RemoveContainer" containerID="89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43" Jul 21 14:35:10.771731 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.771701 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43"} err="failed to get container status \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": rpc error: code = NotFound desc = could not find container \"89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43\": container with ID starting with 89f9caceee67bf6ca00cd608b67c1331e09b46bd93c150c00c898da92a2dea43 not found: ID does not exist" Jul 21 14:35:10.772548 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.772531 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 14:35:10.778603 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.778584 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:35:10.876564 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876529 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876564 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876746 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876746 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876746 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876633 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-config\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876746 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876746 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876746 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qsv\" (UniqueName: \"kubernetes.io/projected/a11c2ef6-45ad-4830-b9d8-2835955866d3-kube-api-access-88qsv\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876860 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a11c2ef6-45ad-4830-b9d8-2835955866d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.876948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a11c2ef6-45ad-4830-b9d8-2835955866d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.877246 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876961 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.877246 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.876979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.877246 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.877017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.877246 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.877043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.916626 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.916595 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18441986-b4c9-40ed-9090-51080bca018f" path="/var/lib/kubelet/pods/18441986-b4c9-40ed-9090-51080bca018f/volumes" Jul 21 14:35:10.978289 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qsv\" (UniqueName: \"kubernetes.io/projected/a11c2ef6-45ad-4830-b9d8-2835955866d3-kube-api-access-88qsv\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978289 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978267 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978468 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978468 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978468 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978468 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978468 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a11c2ef6-45ad-4830-b9d8-2835955866d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978468 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a11c2ef6-45ad-4830-b9d8-2835955866d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978680 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978680 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978680 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978680 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978839 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978697 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978839 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978839 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978839 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.978839 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-config\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979063 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979063 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979063 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979063 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.978999 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979289 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.979091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979289 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.979192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.979291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.979365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.979903 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.979831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.981049 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.981025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.981317 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.981289 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a11c2ef6-45ad-4830-b9d8-2835955866d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.981417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.981335 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a11c2ef6-45ad-4830-b9d8-2835955866d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.981681 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.981650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.982053 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.982029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.982504 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.982440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.982813 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.982791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.983159 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.983140 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a11c2ef6-45ad-4830-b9d8-2835955866d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.983562 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.983540 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-config\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.983684 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.983666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.983854 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.983838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.984231 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.984200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.984326 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.984311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a11c2ef6-45ad-4830-b9d8-2835955866d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:10.986406 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:10.986390 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qsv\" (UniqueName: \"kubernetes.io/projected/a11c2ef6-45ad-4830-b9d8-2835955866d3-kube-api-access-88qsv\") pod \"prometheus-k8s-0\" (UID: \"a11c2ef6-45ad-4830-b9d8-2835955866d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:11.076487 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:11.076461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:11.223576 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:11.223535 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 14:35:11.226717 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:35:11.226691 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11c2ef6_45ad_4830_b9d8_2835955866d3.slice/crio-78ad977ce9fd532492867c3abb41f023338b3750c0c8a561bbd487ccd54be077 WatchSource:0}: Error finding container 78ad977ce9fd532492867c3abb41f023338b3750c0c8a561bbd487ccd54be077: Status 404 returned error can't find the container with id 78ad977ce9fd532492867c3abb41f023338b3750c0c8a561bbd487ccd54be077 Jul 21 14:35:11.710873 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:11.710837 2559 generic.go:356] "Generic (PLEG): container finished" podID="a11c2ef6-45ad-4830-b9d8-2835955866d3" containerID="1554511db7b8d8d6d3b469ba3eed9b0b3987e3051bbac1d317afcad88c79c190" exitCode=0 Jul 21 14:35:11.711343 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:11.710901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerDied","Data":"1554511db7b8d8d6d3b469ba3eed9b0b3987e3051bbac1d317afcad88c79c190"} Jul 21 14:35:11.711343 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:11.711010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerStarted","Data":"78ad977ce9fd532492867c3abb41f023338b3750c0c8a561bbd487ccd54be077"} Jul 21 14:35:12.718069 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:12.718030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerStarted","Data":"192279bf39166926747ca67f6c53b7934c5166f52530d89dcfb304627c2f5443"} Jul 21 14:35:12.718514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:12.718094 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerStarted","Data":"dba49f341bba1a24de6747ef87bc3612fa6e1da5c3f8f98c2ee8e86f526cdb3e"} Jul 21 14:35:12.718514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:12.718119 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerStarted","Data":"85937d9b278a46a5da77959ffa214da88afeabc34dd0411a586c97369c433c8b"} Jul 21 14:35:12.718514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:12.718149 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerStarted","Data":"71fdb1e3c3d9c11dc3853c1bb891840bebd74202756e55ec0a2304c36c8cdb34"} Jul 21 14:35:12.718514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:12.718172 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerStarted","Data":"12f914a095c7909524b0cd5063ffe856a458f915cbbf5b782468fbb4cb142dde"} Jul 21 14:35:12.718514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:12.718194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a11c2ef6-45ad-4830-b9d8-2835955866d3","Type":"ContainerStarted","Data":"b448dfd0b9d4b1eb8777241dcaafb66ead89b6ef04bb7e6b9fecc5382f3d7607"} Jul 21 14:35:12.749188 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:12.749132 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.727122155 podStartE2EDuration="2.749113564s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.991409ms" isStatefulPod=false podCreationTimestamp="2026-07-21 14:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:35:12.74857799 +0000 UTC m=+157.236769231" watchObservedRunningTime="2026-07-21 14:35:12.749113564 +0000 UTC m=+157.237304770" Jul 21 14:35:13.375162 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:13.375131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:14.916816 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:14.916783 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:15.017662 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:15.017617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:15.118539 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:15.118500 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:16.077481 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:16.077434 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:35:20.108486 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:20.108446 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:20.208719 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:20.208672 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:20.308827 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:20.308797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:20.637426 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:20.637392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:26.344822 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:26.344787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:33.802374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:33.802335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:35:33.902186 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:35:33.902148 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:11.076742 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:11.076700 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:36:11.092295 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:11.092247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:36:11.974834 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:11.974807 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 14:36:24.018873 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:24.018836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:24.094202 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:24.094157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:38.839050 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:38.839019 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:50.733445 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:50.733405 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:51.704743 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:51.704713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:54.344899 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:54.344843 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:54.448437 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:54.448405 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:57.593179 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:57.593103 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:57.799494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:57.799448 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:59.969994 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:36:59.969960 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:00.137681 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.137650 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:00.234266 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.234156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:00.244944 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.244917 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2"] Jul 21 14:37:00.249107 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.249073 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.252170 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.252147 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-68497\"" Jul 21 14:37:00.252170 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.252162 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 14:37:00.253449 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.253429 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 14:37:00.255403 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.255384 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2"] Jul 21 14:37:00.318559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.318519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzsnh\" (UniqueName: \"kubernetes.io/projected/94d74fe7-72bb-4eed-be49-77169bbb3e1e-kube-api-access-rzsnh\") pod \"cert-manager-cainjector-559cdd8c87-2gxr2\" (UID: \"94d74fe7-72bb-4eed-be49-77169bbb3e1e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.318559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.318559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d74fe7-72bb-4eed-be49-77169bbb3e1e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2gxr2\" (UID: \"94d74fe7-72bb-4eed-be49-77169bbb3e1e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.418999 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.418959 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzsnh\" (UniqueName: \"kubernetes.io/projected/94d74fe7-72bb-4eed-be49-77169bbb3e1e-kube-api-access-rzsnh\") pod \"cert-manager-cainjector-559cdd8c87-2gxr2\" (UID: \"94d74fe7-72bb-4eed-be49-77169bbb3e1e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.418999 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.419000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d74fe7-72bb-4eed-be49-77169bbb3e1e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2gxr2\" (UID: \"94d74fe7-72bb-4eed-be49-77169bbb3e1e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.427779 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.427749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d74fe7-72bb-4eed-be49-77169bbb3e1e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2gxr2\" (UID: \"94d74fe7-72bb-4eed-be49-77169bbb3e1e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.427891 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.427782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzsnh\" (UniqueName: \"kubernetes.io/projected/94d74fe7-72bb-4eed-be49-77169bbb3e1e-kube-api-access-rzsnh\") pod \"cert-manager-cainjector-559cdd8c87-2gxr2\" (UID: \"94d74fe7-72bb-4eed-be49-77169bbb3e1e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.570132 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.570086 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" Jul 21 14:37:00.698496 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:00.698411 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2"] Jul 21 14:37:00.701250 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:37:00.701204 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94d74fe7_72bb_4eed_be49_77169bbb3e1e.slice/crio-024330f1423fccb5efc98cb958231857dadafe440651077b63fbe1a146759cbb WatchSource:0}: Error finding container 024330f1423fccb5efc98cb958231857dadafe440651077b63fbe1a146759cbb: Status 404 returned error can't find the container with id 024330f1423fccb5efc98cb958231857dadafe440651077b63fbe1a146759cbb Jul 21 14:37:01.162944 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:01.162909 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" event={"ID":"94d74fe7-72bb-4eed-be49-77169bbb3e1e","Type":"ContainerStarted","Data":"024330f1423fccb5efc98cb958231857dadafe440651077b63fbe1a146759cbb"} Jul 21 14:37:07.776596 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:07.776557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:08.194719 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.194684 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" event={"ID":"94d74fe7-72bb-4eed-be49-77169bbb3e1e","Type":"ContainerStarted","Data":"701d2f631f3bb56fd0d07b29b946c02cc0c43cf207a8fbe97d0f473cd0490db6"} Jul 21 14:37:08.211125 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.211075 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2gxr2" podStartSLOduration=0.788692948 podStartE2EDuration="8.211059589s" totalImagesPullingTime="7.422366641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:37:08.208577147 +0000 UTC m=+272.696768393" watchObservedRunningTime="2026-07-21 14:37:08.211059589 +0000 UTC m=+272.699250793" Jul 21 14:37:08.403383 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.403303 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-njzq5"] Jul 21 14:37:08.406585 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.406568 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.409040 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.409015 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-66msk\"" Jul 21 14:37:08.413815 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.413787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-njzq5"] Jul 21 14:37:08.494788 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.494749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zgp\" (UniqueName: \"kubernetes.io/projected/9df3529c-bd9c-413c-b31e-8128c6889e8e-kube-api-access-65zgp\") pod \"cert-manager-55b4d954d5-njzq5\" (UID: \"9df3529c-bd9c-413c-b31e-8128c6889e8e\") " pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.494788 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.494790 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9df3529c-bd9c-413c-b31e-8128c6889e8e-bound-sa-token\") pod \"cert-manager-55b4d954d5-njzq5\" (UID: \"9df3529c-bd9c-413c-b31e-8128c6889e8e\") " pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.596141 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.596086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zgp\" (UniqueName: \"kubernetes.io/projected/9df3529c-bd9c-413c-b31e-8128c6889e8e-kube-api-access-65zgp\") pod \"cert-manager-55b4d954d5-njzq5\" (UID: \"9df3529c-bd9c-413c-b31e-8128c6889e8e\") " pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.596141 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.596143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9df3529c-bd9c-413c-b31e-8128c6889e8e-bound-sa-token\") pod \"cert-manager-55b4d954d5-njzq5\" (UID: \"9df3529c-bd9c-413c-b31e-8128c6889e8e\") " pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.604627 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.604596 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9df3529c-bd9c-413c-b31e-8128c6889e8e-bound-sa-token\") pod \"cert-manager-55b4d954d5-njzq5\" (UID: \"9df3529c-bd9c-413c-b31e-8128c6889e8e\") " pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.604894 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.604872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zgp\" (UniqueName: \"kubernetes.io/projected/9df3529c-bd9c-413c-b31e-8128c6889e8e-kube-api-access-65zgp\") pod \"cert-manager-55b4d954d5-njzq5\" (UID: \"9df3529c-bd9c-413c-b31e-8128c6889e8e\") " pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.716609 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.716515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-njzq5" Jul 21 14:37:08.845499 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.845468 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-njzq5"] Jul 21 14:37:08.849202 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:37:08.849180 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9df3529c_bd9c_413c_b31e_8128c6889e8e.slice/crio-54855de2d5c3785b6196beaad48afc0d930cb69b940454e200fbba9dc5645729 WatchSource:0}: Error finding container 54855de2d5c3785b6196beaad48afc0d930cb69b940454e200fbba9dc5645729: Status 404 returned error can't find the container with id 54855de2d5c3785b6196beaad48afc0d930cb69b940454e200fbba9dc5645729 Jul 21 14:37:08.911415 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:08.911382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:09.011025 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:09.010941 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:09.200402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:09.200363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-njzq5" event={"ID":"9df3529c-bd9c-413c-b31e-8128c6889e8e","Type":"ContainerStarted","Data":"b05bfdbbdd12e8f959f336f9e6e793399888ab3d4eb7db8e1552fe44f9052716"} Jul 21 14:37:09.200583 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:09.200413 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-njzq5" event={"ID":"9df3529c-bd9c-413c-b31e-8128c6889e8e","Type":"ContainerStarted","Data":"54855de2d5c3785b6196beaad48afc0d930cb69b940454e200fbba9dc5645729"} Jul 21 14:37:09.217632 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:09.217582 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-njzq5" podStartSLOduration=1.217563713 podStartE2EDuration="1.217563713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:37:09.216461288 +0000 UTC m=+273.704652540" watchObservedRunningTime="2026-07-21 14:37:09.217563713 +0000 UTC m=+273.705754916" Jul 21 14:37:09.633960 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:09.633920 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:09.724694 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:09.724655 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:15.257317 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:15.257279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:19.862105 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:19.862062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:20.458279 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:20.458242 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:20.514588 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:20.514551 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:34.845567 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:34.845533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:34.940131 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:34.940096 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:35.511770 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:35.511741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.060704 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.060661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.242756 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.242716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.256817 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.256783 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5"] Jul 21 14:37:36.263577 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.263544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.266419 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.266392 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2tzgd\"" Jul 21 14:37:36.266959 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.266941 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 14:37:36.267053 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.267035 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 14:37:36.267656 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.267641 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 14:37:36.267730 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.267713 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 14:37:36.284963 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.284940 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5"] Jul 21 14:37:36.334146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.334062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1525bf0-4f39-42dd-884a-bcff30e78d12-webhook-cert\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.334146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.334102 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhf9\" (UniqueName: \"kubernetes.io/projected/f1525bf0-4f39-42dd-884a-bcff30e78d12-kube-api-access-qnhf9\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.334146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.334125 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1525bf0-4f39-42dd-884a-bcff30e78d12-tmp\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.334146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.334141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1525bf0-4f39-42dd-884a-bcff30e78d12-manifests\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.334493 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.334198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1525bf0-4f39-42dd-884a-bcff30e78d12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.402827 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.402794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.435002 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.434975 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1525bf0-4f39-42dd-884a-bcff30e78d12-webhook-cert\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.435134 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.435017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhf9\" (UniqueName: \"kubernetes.io/projected/f1525bf0-4f39-42dd-884a-bcff30e78d12-kube-api-access-qnhf9\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.435134 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.435051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1525bf0-4f39-42dd-884a-bcff30e78d12-tmp\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.435134 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.435076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1525bf0-4f39-42dd-884a-bcff30e78d12-manifests\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.435134 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.435101 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1525bf0-4f39-42dd-884a-bcff30e78d12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.435593 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.435565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1525bf0-4f39-42dd-884a-bcff30e78d12-manifests\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.437402 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.437375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1525bf0-4f39-42dd-884a-bcff30e78d12-tmp\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.437611 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.437594 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1525bf0-4f39-42dd-884a-bcff30e78d12-webhook-cert\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.437681 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.437664 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f1525bf0-4f39-42dd-884a-bcff30e78d12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.444176 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.444155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhf9\" (UniqueName: \"kubernetes.io/projected/f1525bf0-4f39-42dd-884a-bcff30e78d12-kube-api-access-qnhf9\") pod \"opendatahub-operator-controller-manager-b978d8c-m8sz5\" (UID: \"f1525bf0-4f39-42dd-884a-bcff30e78d12\") " pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.504761 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.504730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.574167 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.574132 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:36.633549 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.633515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.723075 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:37:36.723043 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1525bf0_4f39_42dd_884a_bcff30e78d12.slice/crio-dc84b67373cbb92df3fff22460deaaab9ff9611c2978467ea756056a60f5ab21 WatchSource:0}: Error finding container dc84b67373cbb92df3fff22460deaaab9ff9611c2978467ea756056a60f5ab21: Status 404 returned error can't find the container with id dc84b67373cbb92df3fff22460deaaab9ff9611c2978467ea756056a60f5ab21 Jul 21 14:37:36.723508 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.723484 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5"] Jul 21 14:37:36.727939 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.727911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.748587 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.748565 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:37:36.751249 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.751203 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:37:36.751525 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.751504 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 14:37:36.790878 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.790844 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.887878 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.887794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:36.987862 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:36.987726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:37.321843 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:37.321806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" event={"ID":"f1525bf0-4f39-42dd-884a-bcff30e78d12","Type":"ContainerStarted","Data":"dc84b67373cbb92df3fff22460deaaab9ff9611c2978467ea756056a60f5ab21"} Jul 21 14:37:40.339555 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:40.339522 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" event={"ID":"f1525bf0-4f39-42dd-884a-bcff30e78d12","Type":"ContainerStarted","Data":"f9ccda718f38a2049650088925694be51c747d1723228cde601263c6c2f3688c"} Jul 21 14:37:41.346036 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:41.345989 2559 generic.go:356] "Generic (PLEG): container finished" podID="f1525bf0-4f39-42dd-884a-bcff30e78d12" containerID="f9ccda718f38a2049650088925694be51c747d1723228cde601263c6c2f3688c" exitCode=0 Jul 21 14:37:41.346527 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:41.346072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" event={"ID":"f1525bf0-4f39-42dd-884a-bcff30e78d12","Type":"ContainerDied","Data":"f9ccda718f38a2049650088925694be51c747d1723228cde601263c6c2f3688c"} Jul 21 14:37:41.346704 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:41.346685 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:37:42.352739 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:42.352701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" event={"ID":"f1525bf0-4f39-42dd-884a-bcff30e78d12","Type":"ContainerStarted","Data":"50c04f7c6103f67a9aa3d6b01bb184d3db9795ef39cc5b8f2f5259dc380dd420"} Jul 21 14:37:42.353113 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:42.352789 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:42.382107 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:42.382061 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" podStartSLOduration=1.8173982359999998 podStartE2EDuration="6.382046337s" totalImagesPullingTime="3.52915506s" totalInitContainerRuntime="1.035493041s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:37:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:37:42.379687035 +0000 UTC m=+306.867878239" watchObservedRunningTime="2026-07-21 14:37:42.382046337 +0000 UTC m=+306.870237542" Jul 21 14:37:45.601150 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:45.601120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:47.026108 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:47.026068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:53.359487 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:53.359450 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b978d8c-m8sz5" Jul 21 14:37:54.139936 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:54.139905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:55.636922 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:55.636888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:55.814775 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:55.814746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:56.358089 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:56.358053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:56.425359 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:56.425322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:56.525835 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:56.525808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:56.661468 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:56.661389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:56.932487 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:56.932407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:58.599964 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:37:58.599926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:03.893537 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.893501 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z"] Jul 21 14:38:03.897163 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.897141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:03.900790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.900752 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 14:38:03.900790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.900752 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 14:38:03.900968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.900873 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 14:38:03.900968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.900905 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-c85s5\"" Jul 21 14:38:03.900968 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.900957 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 14:38:03.901106 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.901015 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:38:03.907663 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.907643 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z"] Jul 21 14:38:03.993038 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.993004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsn9\" (UniqueName: \"kubernetes.io/projected/f75dd4fa-b5ce-4576-a202-54038ecce86e-kube-api-access-nxsn9\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:03.993186 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.993059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f75dd4fa-b5ce-4576-a202-54038ecce86e-cert\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:03.993186 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.993155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f75dd4fa-b5ce-4576-a202-54038ecce86e-metrics-cert\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:03.993325 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:03.993203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f75dd4fa-b5ce-4576-a202-54038ecce86e-manager-config\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.093930 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.093896 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f75dd4fa-b5ce-4576-a202-54038ecce86e-manager-config\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.094079 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.093944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsn9\" (UniqueName: \"kubernetes.io/projected/f75dd4fa-b5ce-4576-a202-54038ecce86e-kube-api-access-nxsn9\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.094079 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.093980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f75dd4fa-b5ce-4576-a202-54038ecce86e-cert\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.094079 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.094040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f75dd4fa-b5ce-4576-a202-54038ecce86e-metrics-cert\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.094573 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.094543 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f75dd4fa-b5ce-4576-a202-54038ecce86e-manager-config\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.096454 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.096421 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f75dd4fa-b5ce-4576-a202-54038ecce86e-cert\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.096602 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.096582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f75dd4fa-b5ce-4576-a202-54038ecce86e-metrics-cert\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.102417 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.102395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsn9\" (UniqueName: \"kubernetes.io/projected/f75dd4fa-b5ce-4576-a202-54038ecce86e-kube-api-access-nxsn9\") pod \"lws-controller-manager-77cc7cb7fc-tqm5z\" (UID: \"f75dd4fa-b5ce-4576-a202-54038ecce86e\") " pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.208061 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.207972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:04.340392 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.340368 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z"] Jul 21 14:38:04.343032 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:38:04.343007 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75dd4fa_b5ce_4576_a202_54038ecce86e.slice/crio-bdae620803d2bdb2fe9b31b82ad539448becc1cdf2d3e998f1424d98a745eedf WatchSource:0}: Error finding container bdae620803d2bdb2fe9b31b82ad539448becc1cdf2d3e998f1424d98a745eedf: Status 404 returned error can't find the container with id bdae620803d2bdb2fe9b31b82ad539448becc1cdf2d3e998f1424d98a745eedf Jul 21 14:38:04.451788 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:04.451753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" event={"ID":"f75dd4fa-b5ce-4576-a202-54038ecce86e","Type":"ContainerStarted","Data":"bdae620803d2bdb2fe9b31b82ad539448becc1cdf2d3e998f1424d98a745eedf"} Jul 21 14:38:05.912829 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:05.912774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:06.012280 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.012249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:06.110270 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.110236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:06.302529 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.302494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:06.628406 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.628309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:06.630086 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.630056 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5"] Jul 21 14:38:06.640847 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.640826 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.646741 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.646330 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5"] Jul 21 14:38:06.647395 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.647373 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 14:38:06.647395 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.647385 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 14:38:06.647619 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.647591 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 14:38:06.647950 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.647749 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 14:38:06.647950 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.647766 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dc4bk\"" Jul 21 14:38:06.719128 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.719090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50731338-efa4-44f8-9c26-cd899e23915a-tmp\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.719326 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.719156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50731338-efa4-44f8-9c26-cd899e23915a-tls-certs\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.719326 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.719296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqpvf\" (UniqueName: \"kubernetes.io/projected/50731338-efa4-44f8-9c26-cd899e23915a-kube-api-access-pqpvf\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.820600 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.820566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50731338-efa4-44f8-9c26-cd899e23915a-tls-certs\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.820786 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.820668 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqpvf\" (UniqueName: \"kubernetes.io/projected/50731338-efa4-44f8-9c26-cd899e23915a-kube-api-access-pqpvf\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.820786 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:38:06.820725 2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 21 14:38:06.820786 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.820735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50731338-efa4-44f8-9c26-cd899e23915a-tmp\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.820942 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:38:06.820807 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50731338-efa4-44f8-9c26-cd899e23915a-tls-certs podName:50731338-efa4-44f8-9c26-cd899e23915a nodeName:}" failed. No retries permitted until 2026-07-21 14:38:07.320787052 +0000 UTC m=+331.808978243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/50731338-efa4-44f8-9c26-cd899e23915a-tls-certs") pod "kube-auth-proxy-74d9d9b8c-j9xq5" (UID: "50731338-efa4-44f8-9c26-cd899e23915a") : secret "kube-auth-proxy-tls" not found Jul 21 14:38:06.822930 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.822895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50731338-efa4-44f8-9c26-cd899e23915a-tmp\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.839896 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.839870 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqpvf\" (UniqueName: \"kubernetes.io/projected/50731338-efa4-44f8-9c26-cd899e23915a-kube-api-access-pqpvf\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:06.944870 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:06.944752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:07.228061 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.228033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:07.327038 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.326999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50731338-efa4-44f8-9c26-cd899e23915a-tls-certs\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:07.329791 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.329766 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50731338-efa4-44f8-9c26-cd899e23915a-tls-certs\") pod \"kube-auth-proxy-74d9d9b8c-j9xq5\" (UID: \"50731338-efa4-44f8-9c26-cd899e23915a\") " pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:07.335655 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.335628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:07.469522 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.469433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" event={"ID":"f75dd4fa-b5ce-4576-a202-54038ecce86e","Type":"ContainerStarted","Data":"981f66e3ce8c37c04068c7f2d379ed2694625b6b62a0a634dedc79a2770001a5"} Jul 21 14:38:07.469682 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.469580 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:07.490642 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.490587 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" podStartSLOduration=1.624844447 podStartE2EDuration="4.490573768s" totalImagesPullingTime="2.865729321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:38:07.48951759 +0000 UTC m=+331.977708799" watchObservedRunningTime="2026-07-21 14:38:07.490573768 +0000 UTC m=+331.978764972" Jul 21 14:38:07.557091 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.557043 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" Jul 21 14:38:07.708456 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:07.708426 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5"] Jul 21 14:38:07.711393 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:38:07.711355 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50731338_efa4_44f8_9c26_cd899e23915a.slice/crio-43dd523d4615a5399569e60335d719f83f6503fde4adbcc290321b41a9ab72dd WatchSource:0}: Error finding container 43dd523d4615a5399569e60335d719f83f6503fde4adbcc290321b41a9ab72dd: Status 404 returned error can't find the container with id 43dd523d4615a5399569e60335d719f83f6503fde4adbcc290321b41a9ab72dd Jul 21 14:38:08.474990 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:08.474956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" event={"ID":"50731338-efa4-44f8-9c26-cd899e23915a","Type":"ContainerStarted","Data":"43dd523d4615a5399569e60335d719f83f6503fde4adbcc290321b41a9ab72dd"} Jul 21 14:38:08.621805 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:08.621773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:08.713080 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:08.713044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:09.843340 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:09.843303 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:11.494069 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:11.494028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" event={"ID":"50731338-efa4-44f8-9c26-cd899e23915a","Type":"ContainerStarted","Data":"ec39dfeec12f507a0c2b3960a8aba177b2278c6ba8e307eb847eca16b4f908f6"} Jul 21 14:38:11.510878 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:11.510829 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74d9d9b8c-j9xq5" podStartSLOduration=2.447533654 podStartE2EDuration="5.510814767s" totalImagesPullingTime="3.063281113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:38:11.510195757 +0000 UTC m=+335.998386964" watchObservedRunningTime="2026-07-21 14:38:11.510814767 +0000 UTC m=+335.999005971" Jul 21 14:38:14.031547 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:14.031508 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:14.361311 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:14.361195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:14.920650 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:14.920608 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:15.021553 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:15.021523 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:18.477600 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:18.477564 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77cc7cb7fc-tqm5z" Jul 21 14:38:57.142401 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:57.142340 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:57.248642 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:57.248602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:57.314347 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:57.314283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:58.062013 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:58.061975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:58.259689 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:38:58.259658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:00.427944 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:00.427906 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:00.564901 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:00.564863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:01.166237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:01.166170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:11.844987 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:11.844954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:11.943773 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:11.943730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:12.009665 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:12.009637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:31.870687 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:31.870649 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:35.068429 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:35.068375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:39.913018 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:39.912983 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:40.015753 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:40.015721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:40.135175 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:40.135129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:40.470684 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:40.470640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:41.915506 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:41.915473 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:43.086374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.086343 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh"] Jul 21 14:39:43.091301 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.091279 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" Jul 21 14:39:43.093939 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.093920 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 14:39:43.094972 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.094943 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 14:39:43.094972 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.094954 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qhvh8\"" Jul 21 14:39:43.099601 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.099575 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh"] Jul 21 14:39:43.204434 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.204400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gflv\" (UniqueName: \"kubernetes.io/projected/482162f6-cb38-460a-bd48-17e37b8e01ed-kube-api-access-6gflv\") pod \"authorino-operator-75cc8d5cb7-b4pzh\" (UID: \"482162f6-cb38-460a-bd48-17e37b8e01ed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" Jul 21 14:39:43.305101 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.305070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gflv\" (UniqueName: \"kubernetes.io/projected/482162f6-cb38-460a-bd48-17e37b8e01ed-kube-api-access-6gflv\") pod \"authorino-operator-75cc8d5cb7-b4pzh\" (UID: \"482162f6-cb38-460a-bd48-17e37b8e01ed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" Jul 21 14:39:43.317563 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.317532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gflv\" (UniqueName: \"kubernetes.io/projected/482162f6-cb38-460a-bd48-17e37b8e01ed-kube-api-access-6gflv\") pod \"authorino-operator-75cc8d5cb7-b4pzh\" (UID: \"482162f6-cb38-460a-bd48-17e37b8e01ed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" Jul 21 14:39:43.402439 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.402345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" Jul 21 14:39:43.536807 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.536776 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh"] Jul 21 14:39:43.540712 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:39:43.540684 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482162f6_cb38_460a_bd48_17e37b8e01ed.slice/crio-f60b48a1c50c1cd36e19fc2075b0dc0dc55a15f0021ba05d8e84b0dcf1011f64 WatchSource:0}: Error finding container f60b48a1c50c1cd36e19fc2075b0dc0dc55a15f0021ba05d8e84b0dcf1011f64: Status 404 returned error can't find the container with id f60b48a1c50c1cd36e19fc2075b0dc0dc55a15f0021ba05d8e84b0dcf1011f64 Jul 21 14:39:43.919074 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:43.919038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" event={"ID":"482162f6-cb38-460a-bd48-17e37b8e01ed","Type":"ContainerStarted","Data":"f60b48a1c50c1cd36e19fc2075b0dc0dc55a15f0021ba05d8e84b0dcf1011f64"} Jul 21 14:39:44.338763 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:44.338732 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:44.526756 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:44.526711 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:46.521521 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:46.521494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:48.327668 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:48.327635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:48.431435 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:48.431397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:48.552753 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:48.552722 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:48.943685 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:48.943597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" event={"ID":"482162f6-cb38-460a-bd48-17e37b8e01ed","Type":"ContainerStarted","Data":"67873ca7dcdc34234f22f73736d0ddcb41bf83e39112eb12ceaf4d21870f0a2d"} Jul 21 14:39:48.943685 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:48.943651 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" Jul 21 14:39:48.968639 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:48.968568 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" podStartSLOduration=1.237480001 podStartE2EDuration="5.968551679s" totalImagesPullingTime="4.731071678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:39:48.966318157 +0000 UTC m=+433.454509384" watchObservedRunningTime="2026-07-21 14:39:48.968551679 +0000 UTC m=+433.456742883" Jul 21 14:39:51.480405 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:51.480373 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:51.580668 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:51.580636 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:39:59.950653 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:39:59.950577 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b4pzh" Jul 21 14:40:00.110979 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:00.110928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:00.215821 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:00.215730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:02.588116 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:02.588084 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:02.684647 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:02.684594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:06.890490 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:06.890460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:06.991242 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:06.991193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:09.733243 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:09.733187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:09.813575 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:09.813542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:18.897937 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:18.897900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:18.998494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:18.998460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:21.344113 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:21.344075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:21.445143 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:21.445111 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:22.236387 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:22.236347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:31.413164 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:31.413130 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:31.516664 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:31.516628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:36.782600 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:36.782569 2559 scope.go:117] "RemoveContainer" containerID="587eb102df1b5d802b5c5cfcafb3a9190c82f5c39f63486d2ccc3c2e67f1b849" Jul 21 14:40:45.795711 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:45.795675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:45.905594 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:45.905564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:46.006300 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.006265 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:46.585829 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.585794 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-nqpfd"] Jul 21 14:40:46.591053 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.591028 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nqpfd" Jul 21 14:40:46.594095 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.594069 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mg9d8\"" Jul 21 14:40:46.594740 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.594711 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-nqpfd"] Jul 21 14:40:46.616552 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.616505 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:46.670621 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.670578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:40:46.677793 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.677771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhjb\" (UniqueName: \"kubernetes.io/projected/e63acc22-dfb3-4f4f-887f-de78832dc4f8-kube-api-access-txhjb\") pod \"authorino-58cd955675-nqpfd\" (UID: \"e63acc22-dfb3-4f4f-887f-de78832dc4f8\") " pod="kuadrant-system/authorino-58cd955675-nqpfd" Jul 21 14:40:46.778482 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.778441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhjb\" (UniqueName: \"kubernetes.io/projected/e63acc22-dfb3-4f4f-887f-de78832dc4f8-kube-api-access-txhjb\") pod \"authorino-58cd955675-nqpfd\" (UID: \"e63acc22-dfb3-4f4f-887f-de78832dc4f8\") " pod="kuadrant-system/authorino-58cd955675-nqpfd" Jul 21 14:40:46.786818 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.786792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhjb\" (UniqueName: \"kubernetes.io/projected/e63acc22-dfb3-4f4f-887f-de78832dc4f8-kube-api-access-txhjb\") pod \"authorino-58cd955675-nqpfd\" (UID: \"e63acc22-dfb3-4f4f-887f-de78832dc4f8\") " pod="kuadrant-system/authorino-58cd955675-nqpfd" Jul 21 14:40:46.901298 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:46.901200 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nqpfd" Jul 21 14:40:47.035969 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:47.035946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-nqpfd"] Jul 21 14:40:47.038372 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:40:47.038346 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode63acc22_dfb3_4f4f_887f_de78832dc4f8.slice/crio-8bd358cfd10fc3b9a4eac2c75c63e8688fab463fa0e60c4f53629ec5bad21c1a WatchSource:0}: Error finding container 8bd358cfd10fc3b9a4eac2c75c63e8688fab463fa0e60c4f53629ec5bad21c1a: Status 404 returned error can't find the container with id 8bd358cfd10fc3b9a4eac2c75c63e8688fab463fa0e60c4f53629ec5bad21c1a Jul 21 14:40:47.222914 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:47.222829 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nqpfd" event={"ID":"e63acc22-dfb3-4f4f-887f-de78832dc4f8","Type":"ContainerStarted","Data":"8bd358cfd10fc3b9a4eac2c75c63e8688fab463fa0e60c4f53629ec5bad21c1a"} Jul 21 14:40:50.242512 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:50.242422 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nqpfd" event={"ID":"e63acc22-dfb3-4f4f-887f-de78832dc4f8","Type":"ContainerStarted","Data":"212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed"} Jul 21 14:40:50.257990 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:50.257941 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-nqpfd" podStartSLOduration=1.319444283 podStartE2EDuration="4.257926637s" totalImagesPullingTime="2.938482354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:40:50.25616918 +0000 UTC m=+494.744360410" watchObservedRunningTime="2026-07-21 14:40:50.257926637 +0000 UTC m=+494.746117841" Jul 21 14:40:51.760732 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:51.760691 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-nqpfd"] Jul 21 14:40:52.251933 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:52.251895 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-nqpfd" podUID="e63acc22-dfb3-4f4f-887f-de78832dc4f8" containerName="authorino" containerID="cri-o://212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed" gracePeriod=30 Jul 21 14:40:52.493107 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:52.493080 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nqpfd" Jul 21 14:40:52.533100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:52.533075 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txhjb\" (UniqueName: \"kubernetes.io/projected/e63acc22-dfb3-4f4f-887f-de78832dc4f8-kube-api-access-txhjb\") pod \"e63acc22-dfb3-4f4f-887f-de78832dc4f8\" (UID: \"e63acc22-dfb3-4f4f-887f-de78832dc4f8\") " Jul 21 14:40:52.535260 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:52.535230 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63acc22-dfb3-4f4f-887f-de78832dc4f8-kube-api-access-txhjb" (OuterVolumeSpecName: "kube-api-access-txhjb") pod "e63acc22-dfb3-4f4f-887f-de78832dc4f8" (UID: "e63acc22-dfb3-4f4f-887f-de78832dc4f8"). InnerVolumeSpecName "kube-api-access-txhjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:40:52.634627 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:52.634586 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txhjb\" (UniqueName: \"kubernetes.io/projected/e63acc22-dfb3-4f4f-887f-de78832dc4f8-kube-api-access-txhjb\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:40:53.257298 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.257185 2559 generic.go:356] "Generic (PLEG): container finished" podID="e63acc22-dfb3-4f4f-887f-de78832dc4f8" containerID="212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed" exitCode=0 Jul 21 14:40:53.257298 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.257271 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-nqpfd" Jul 21 14:40:53.257710 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.257263 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nqpfd" event={"ID":"e63acc22-dfb3-4f4f-887f-de78832dc4f8","Type":"ContainerDied","Data":"212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed"} Jul 21 14:40:53.257710 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.257380 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-nqpfd" event={"ID":"e63acc22-dfb3-4f4f-887f-de78832dc4f8","Type":"ContainerDied","Data":"8bd358cfd10fc3b9a4eac2c75c63e8688fab463fa0e60c4f53629ec5bad21c1a"} Jul 21 14:40:53.257710 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.257397 2559 scope.go:117] "RemoveContainer" containerID="212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed" Jul 21 14:40:53.265802 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.265780 2559 scope.go:117] "RemoveContainer" containerID="212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed" Jul 21 14:40:53.266072 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:40:53.266053 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed\": container with ID starting with 212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed not found: ID does not exist" containerID="212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed" Jul 21 14:40:53.266119 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.266081 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed"} err="failed to get container status \"212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed\": rpc error: code = NotFound desc = could not find container \"212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed\": container with ID starting with 212d04f3e9dbaae1b6d89db3709bf70df584d219c9bef584973b0997220a1bed not found: ID does not exist" Jul 21 14:40:53.273701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.273679 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-nqpfd"] Jul 21 14:40:53.275156 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:53.275137 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-nqpfd"] Jul 21 14:40:54.916255 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:40:54.916206 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e63acc22-dfb3-4f4f-887f-de78832dc4f8" path="/var/lib/kubelet/pods/e63acc22-dfb3-4f4f-887f-de78832dc4f8/volumes" Jul 21 14:41:05.361874 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:05.361831 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:05.474069 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:05.474035 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:05.874604 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:05.874571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:18.602891 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.602852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:18.837412 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.837378 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-cz8vp"] Jul 21 14:41:18.837801 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.837788 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63acc22-dfb3-4f4f-887f-de78832dc4f8" containerName="authorino" Jul 21 14:41:18.837846 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.837802 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63acc22-dfb3-4f4f-887f-de78832dc4f8" containerName="authorino" Jul 21 14:41:18.837891 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.837867 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e63acc22-dfb3-4f4f-887f-de78832dc4f8" containerName="authorino" Jul 21 14:41:18.842122 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.842102 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-cz8vp" Jul 21 14:41:18.844597 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.844574 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mg9d8\"" Jul 21 14:41:18.847750 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.847724 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-cz8vp"] Jul 21 14:41:18.911467 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.911379 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:18.971787 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:18.971758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkxlq\" (UniqueName: \"kubernetes.io/projected/03d64d5f-ea9c-4aa4-815d-368955251e3e-kube-api-access-dkxlq\") pod \"authorino-5d4b67c446-cz8vp\" (UID: \"03d64d5f-ea9c-4aa4-815d-368955251e3e\") " pod="kuadrant-system/authorino-5d4b67c446-cz8vp" Jul 21 14:41:19.071816 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.071786 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-cz8vp"] Jul 21 14:41:19.072064 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:41:19.072045 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dkxlq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-cz8vp" podUID="03d64d5f-ea9c-4aa4-815d-368955251e3e" Jul 21 14:41:19.072320 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.072305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkxlq\" (UniqueName: \"kubernetes.io/projected/03d64d5f-ea9c-4aa4-815d-368955251e3e-kube-api-access-dkxlq\") pod \"authorino-5d4b67c446-cz8vp\" (UID: \"03d64d5f-ea9c-4aa4-815d-368955251e3e\") " pod="kuadrant-system/authorino-5d4b67c446-cz8vp" Jul 21 14:41:19.083397 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.083369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkxlq\" (UniqueName: \"kubernetes.io/projected/03d64d5f-ea9c-4aa4-815d-368955251e3e-kube-api-access-dkxlq\") pod \"authorino-5d4b67c446-cz8vp\" (UID: \"03d64d5f-ea9c-4aa4-815d-368955251e3e\") " pod="kuadrant-system/authorino-5d4b67c446-cz8vp" Jul 21 14:41:19.096298 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.096273 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c4586f57-4h6wp"] Jul 21 14:41:19.098722 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.098708 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4586f57-4h6wp" Jul 21 14:41:19.112678 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.112654 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c4586f57-4h6wp"] Jul 21 14:41:19.190407 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.190326 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c4586f57-4h6wp"] Jul 21 14:41:19.190616 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:41:19.190595 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-94gsc], unattached volumes=[], failed to process volumes=[kube-api-access-94gsc]: context canceled" pod="kuadrant-system/authorino-5c4586f57-4h6wp" podUID="c24d658e-86d2-4efb-9981-d752b73b35c8" Jul 21 14:41:19.223147 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.223114 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b8c55c54f-k8h6r"] Jul 21 14:41:19.225651 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.225631 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.228119 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.228102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 14:41:19.233605 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.233584 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b8c55c54f-k8h6r"] Jul 21 14:41:19.274169 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.274142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94gsc\" (UniqueName: \"kubernetes.io/projected/c24d658e-86d2-4efb-9981-d752b73b35c8-kube-api-access-94gsc\") pod \"authorino-5c4586f57-4h6wp\" (UID: \"c24d658e-86d2-4efb-9981-d752b73b35c8\") " pod="kuadrant-system/authorino-5c4586f57-4h6wp" Jul 21 14:41:19.375500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.375462 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a21e448-f731-4cad-8ece-bdd4e34bf7df-tls-cert\") pod \"authorino-b8c55c54f-k8h6r\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.375662 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.375566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94gsc\" (UniqueName: \"kubernetes.io/projected/c24d658e-86d2-4efb-9981-d752b73b35c8-kube-api-access-94gsc\") pod \"authorino-5c4586f57-4h6wp\" (UID: \"c24d658e-86d2-4efb-9981-d752b73b35c8\") " pod="kuadrant-system/authorino-5c4586f57-4h6wp" Jul 21 14:41:19.375662 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.375589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68sd9\" (UniqueName: \"kubernetes.io/projected/6a21e448-f731-4cad-8ece-bdd4e34bf7df-kube-api-access-68sd9\") pod \"authorino-b8c55c54f-k8h6r\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.383882 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.383857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94gsc\" (UniqueName: \"kubernetes.io/projected/c24d658e-86d2-4efb-9981-d752b73b35c8-kube-api-access-94gsc\") pod \"authorino-5c4586f57-4h6wp\" (UID: \"c24d658e-86d2-4efb-9981-d752b73b35c8\") " pod="kuadrant-system/authorino-5c4586f57-4h6wp" Jul 21 14:41:19.385543 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.385514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-cz8vp" Jul 21 14:41:19.385635 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.385514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4586f57-4h6wp" Jul 21 14:41:19.396501 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.396481 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-cz8vp" Jul 21 14:41:19.399875 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.399856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4586f57-4h6wp" Jul 21 14:41:19.476629 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.476527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a21e448-f731-4cad-8ece-bdd4e34bf7df-tls-cert\") pod \"authorino-b8c55c54f-k8h6r\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.476790 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.476659 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68sd9\" (UniqueName: \"kubernetes.io/projected/6a21e448-f731-4cad-8ece-bdd4e34bf7df-kube-api-access-68sd9\") pod \"authorino-b8c55c54f-k8h6r\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.478996 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.478975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a21e448-f731-4cad-8ece-bdd4e34bf7df-tls-cert\") pod \"authorino-b8c55c54f-k8h6r\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.485702 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.485680 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68sd9\" (UniqueName: \"kubernetes.io/projected/6a21e448-f731-4cad-8ece-bdd4e34bf7df-kube-api-access-68sd9\") pod \"authorino-b8c55c54f-k8h6r\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.535264 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.535231 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:41:19.577479 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.577447 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkxlq\" (UniqueName: \"kubernetes.io/projected/03d64d5f-ea9c-4aa4-815d-368955251e3e-kube-api-access-dkxlq\") pod \"03d64d5f-ea9c-4aa4-815d-368955251e3e\" (UID: \"03d64d5f-ea9c-4aa4-815d-368955251e3e\") " Jul 21 14:41:19.577676 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.577500 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94gsc\" (UniqueName: \"kubernetes.io/projected/c24d658e-86d2-4efb-9981-d752b73b35c8-kube-api-access-94gsc\") pod \"c24d658e-86d2-4efb-9981-d752b73b35c8\" (UID: \"c24d658e-86d2-4efb-9981-d752b73b35c8\") " Jul 21 14:41:19.579559 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.579527 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24d658e-86d2-4efb-9981-d752b73b35c8-kube-api-access-94gsc" (OuterVolumeSpecName: "kube-api-access-94gsc") pod "c24d658e-86d2-4efb-9981-d752b73b35c8" (UID: "c24d658e-86d2-4efb-9981-d752b73b35c8"). InnerVolumeSpecName "kube-api-access-94gsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:41:19.579697 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.579612 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d64d5f-ea9c-4aa4-815d-368955251e3e-kube-api-access-dkxlq" (OuterVolumeSpecName: "kube-api-access-dkxlq") pod "03d64d5f-ea9c-4aa4-815d-368955251e3e" (UID: "03d64d5f-ea9c-4aa4-815d-368955251e3e"). InnerVolumeSpecName "kube-api-access-dkxlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:41:19.662835 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.662812 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b8c55c54f-k8h6r"] Jul 21 14:41:19.665561 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:41:19.665530 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a21e448_f731_4cad_8ece_bdd4e34bf7df.slice/crio-c981f779a2d51eed40dd531beed8b035707b41ddd95d3b223137cdc07cdff36a WatchSource:0}: Error finding container c981f779a2d51eed40dd531beed8b035707b41ddd95d3b223137cdc07cdff36a: Status 404 returned error can't find the container with id c981f779a2d51eed40dd531beed8b035707b41ddd95d3b223137cdc07cdff36a Jul 21 14:41:19.678322 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.678296 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkxlq\" (UniqueName: \"kubernetes.io/projected/03d64d5f-ea9c-4aa4-815d-368955251e3e-kube-api-access-dkxlq\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:41:19.678322 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:19.678323 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94gsc\" (UniqueName: \"kubernetes.io/projected/c24d658e-86d2-4efb-9981-d752b73b35c8-kube-api-access-94gsc\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:41:20.390536 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.390455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-cz8vp" Jul 21 14:41:20.390715 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.390443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" event={"ID":"6a21e448-f731-4cad-8ece-bdd4e34bf7df","Type":"ContainerStarted","Data":"c981f779a2d51eed40dd531beed8b035707b41ddd95d3b223137cdc07cdff36a"} Jul 21 14:41:20.390715 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.390455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4586f57-4h6wp" Jul 21 14:41:20.422265 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.422238 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c4586f57-4h6wp"] Jul 21 14:41:20.426403 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.426376 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c4586f57-4h6wp"] Jul 21 14:41:20.449880 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.449850 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-cz8vp"] Jul 21 14:41:20.454571 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.454537 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-cz8vp"] Jul 21 14:41:20.917446 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.917401 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d64d5f-ea9c-4aa4-815d-368955251e3e" path="/var/lib/kubelet/pods/03d64d5f-ea9c-4aa4-815d-368955251e3e/volumes" Jul 21 14:41:20.918023 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:20.917795 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c24d658e-86d2-4efb-9981-d752b73b35c8" path="/var/lib/kubelet/pods/c24d658e-86d2-4efb-9981-d752b73b35c8/volumes" Jul 21 14:41:21.398192 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:21.398155 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" event={"ID":"6a21e448-f731-4cad-8ece-bdd4e34bf7df","Type":"ContainerStarted","Data":"6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b"} Jul 21 14:41:21.415976 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:21.415924 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" podStartSLOduration=0.80431008 podStartE2EDuration="2.415909392s" totalImagesPullingTime="1.611599312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:41:21.413971286 +0000 UTC m=+525.902162512" watchObservedRunningTime="2026-07-21 14:41:21.415909392 +0000 UTC m=+525.904100596" Jul 21 14:41:27.655696 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:27.655609 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:27.748917 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:27.748876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:27.802443 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:27.802411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:30.706765 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:30.706732 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:30.805055 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:30.805012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:49.029588 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:41:49.029553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:00.248498 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:00.248457 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:00.348182 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:00.348132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:00.928453 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:00.928421 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:01.200952 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.200859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:01.459873 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.459784 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:01.557766 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.557726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:01.574294 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.574256 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-s4tx5"] Jul 21 14:42:01.577856 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.577839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:01.580982 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.580961 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 14:42:01.581123 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.581087 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gp6pj\"" Jul 21 14:42:01.586066 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.586022 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-s4tx5"] Jul 21 14:42:01.760921 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.760822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cae0ef-d6e1-4d17-abbd-16dfdfda4a73-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-s4tx5\" (UID: \"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:01.761109 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.761034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66z8d\" (UniqueName: \"kubernetes.io/projected/64cae0ef-d6e1-4d17-abbd-16dfdfda4a73-kube-api-access-66z8d\") pod \"payload-pre-processing-7677b774bb-s4tx5\" (UID: \"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:01.862076 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.862036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66z8d\" (UniqueName: \"kubernetes.io/projected/64cae0ef-d6e1-4d17-abbd-16dfdfda4a73-kube-api-access-66z8d\") pod \"payload-pre-processing-7677b774bb-s4tx5\" (UID: \"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:01.862293 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.862094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cae0ef-d6e1-4d17-abbd-16dfdfda4a73-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-s4tx5\" (UID: \"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:01.862804 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.862786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cae0ef-d6e1-4d17-abbd-16dfdfda4a73-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-s4tx5\" (UID: \"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:01.872407 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.872372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66z8d\" (UniqueName: \"kubernetes.io/projected/64cae0ef-d6e1-4d17-abbd-16dfdfda4a73-kube-api-access-66z8d\") pod \"payload-pre-processing-7677b774bb-s4tx5\" (UID: \"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:01.889953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:01.889908 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:02.023461 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:02.023436 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-s4tx5"] Jul 21 14:42:02.025755 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:42:02.025728 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cae0ef_d6e1_4d17_abbd_16dfdfda4a73.slice/crio-2dbe9dcc4865efb6047d3dab5113b76169915890c9cdec9933af07ac08ed5321 WatchSource:0}: Error finding container 2dbe9dcc4865efb6047d3dab5113b76169915890c9cdec9933af07ac08ed5321: Status 404 returned error can't find the container with id 2dbe9dcc4865efb6047d3dab5113b76169915890c9cdec9933af07ac08ed5321 Jul 21 14:42:02.601644 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:02.601600 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" event={"ID":"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73","Type":"ContainerStarted","Data":"2dbe9dcc4865efb6047d3dab5113b76169915890c9cdec9933af07ac08ed5321"} Jul 21 14:42:04.340832 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:04.340785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:04.398662 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:04.398629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:04.612282 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:04.612166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" event={"ID":"64cae0ef-d6e1-4d17-abbd-16dfdfda4a73","Type":"ContainerStarted","Data":"21d0ed2091efe0c10f5ab852b6aa4b22fef7c76f1365fa1412c6c81533e07bc2"} Jul 21 14:42:04.612282 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:04.612268 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:10.473937 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:10.473905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:10.574344 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:10.574314 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:15.619629 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:15.619589 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" Jul 21 14:42:15.637049 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:15.636996 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-s4tx5" podStartSLOduration=13.083865904 podStartE2EDuration="14.636978819s" totalImagesPullingTime="1.553112915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:42:04.626637886 +0000 UTC m=+569.114829092" watchObservedRunningTime="2026-07-21 14:42:15.636978819 +0000 UTC m=+580.125170023" Jul 21 14:42:15.740540 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:15.740505 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:18.031793 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:18.031750 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:27.444509 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:27.444468 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:28.579500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.579463 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x"] Jul 21 14:42:28.583229 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.583198 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.585725 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.585703 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 14:42:28.586013 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.585997 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 14:42:28.586069 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.586013 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 14:42:28.587117 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.587098 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pq8h2\"" Jul 21 14:42:28.594701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.594674 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x"] Jul 21 14:42:28.668272 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.668238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.668458 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.668347 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.668458 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.668398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.668458 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.668438 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ecc370-f3c6-4120-a9e6-402a6d5c920b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.668599 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.668461 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.668599 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.668484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz9m\" (UniqueName: \"kubernetes.io/projected/49ecc370-f3c6-4120-a9e6-402a6d5c920b-kube-api-access-zvz9m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769328 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ecc370-f3c6-4120-a9e6-402a6d5c920b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz9m\" (UniqueName: \"kubernetes.io/projected/49ecc370-f3c6-4120-a9e6-402a6d5c920b-kube-api-access-zvz9m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769500 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769762 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769733 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769825 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.769869 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.769832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.771686 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.771665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ecc370-f3c6-4120-a9e6-402a6d5c920b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.771849 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.771832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ecc370-f3c6-4120-a9e6-402a6d5c920b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.777322 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.777293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz9m\" (UniqueName: \"kubernetes.io/projected/49ecc370-f3c6-4120-a9e6-402a6d5c920b-kube-api-access-zvz9m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lrs6x\" (UID: \"49ecc370-f3c6-4120-a9e6-402a6d5c920b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.895558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.895453 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:28.901927 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:28.901880 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:29.032872 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:29.032843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x"] Jul 21 14:42:29.035058 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:42:29.035031 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ecc370_f3c6_4120_a9e6_402a6d5c920b.slice/crio-3bd83cbc990fa6b72831966d44993f10ae1a53994b4141443756045f4b41f6d0 WatchSource:0}: Error finding container 3bd83cbc990fa6b72831966d44993f10ae1a53994b4141443756045f4b41f6d0: Status 404 returned error can't find the container with id 3bd83cbc990fa6b72831966d44993f10ae1a53994b4141443756045f4b41f6d0 Jul 21 14:42:29.742689 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:29.742630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" event={"ID":"49ecc370-f3c6-4120-a9e6-402a6d5c920b","Type":"ContainerStarted","Data":"3bd83cbc990fa6b72831966d44993f10ae1a53994b4141443756045f4b41f6d0"} Jul 21 14:42:30.748395 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:30.748350 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" event={"ID":"49ecc370-f3c6-4120-a9e6-402a6d5c920b","Type":"ContainerStarted","Data":"1792eebd3b9238f7d338a30d9607371bb6de0f596da8bb0f8759136f7fdd5e4b"} Jul 21 14:42:30.766843 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:30.766791 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" podStartSLOduration=1.624636151 podStartE2EDuration="2.766775521s" totalImagesPullingTime="1.14213937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:42:30.766031951 +0000 UTC m=+595.254223156" watchObservedRunningTime="2026-07-21 14:42:30.766775521 +0000 UTC m=+595.254966737" Jul 21 14:42:35.216918 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:35.216887 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:36.786003 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:36.785972 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:42:36.786475 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:36.786295 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:42:38.896496 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:38.896464 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:38.898570 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:38.896513 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:38.909433 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:38.909408 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:39.800379 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:39.800350 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lrs6x" Jul 21 14:42:46.672522 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.672491 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb"] Jul 21 14:42:46.675916 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.675894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.678479 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.678459 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 14:42:46.685128 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.685108 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb"] Jul 21 14:42:46.709698 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.709671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:46.730032 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.730009 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtgw\" (UniqueName: \"kubernetes.io/projected/302bf22c-580e-4cbb-8dff-542048009696-kube-api-access-8mtgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.730113 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.730038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/302bf22c-580e-4cbb-8dff-542048009696-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.730113 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.730058 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.730198 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.730112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.730198 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.730185 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.730299 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.730270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.831705 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.831672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.831896 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.831717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtgw\" (UniqueName: \"kubernetes.io/projected/302bf22c-580e-4cbb-8dff-542048009696-kube-api-access-8mtgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.831896 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.831739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/302bf22c-580e-4cbb-8dff-542048009696-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.831896 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.831758 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.831896 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.831780 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.831896 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.831822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.832282 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.832255 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.832282 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.832272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.832422 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.832332 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.834022 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.833997 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/302bf22c-580e-4cbb-8dff-542048009696-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.834271 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.834255 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/302bf22c-580e-4cbb-8dff-542048009696-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.839331 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.839311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtgw\" (UniqueName: \"kubernetes.io/projected/302bf22c-580e-4cbb-8dff-542048009696-kube-api-access-8mtgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb\" (UID: \"302bf22c-580e-4cbb-8dff-542048009696\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:46.986397 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:46.986305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:47.122771 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:47.122745 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb"] Jul 21 14:42:47.124839 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:42:47.124803 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302bf22c_580e_4cbb_8dff_542048009696.slice/crio-11536d7917abd7187739ff5fa0cc09a89d6cc483378f4691fd2feed1f4bbeaf9 WatchSource:0}: Error finding container 11536d7917abd7187739ff5fa0cc09a89d6cc483378f4691fd2feed1f4bbeaf9: Status 404 returned error can't find the container with id 11536d7917abd7187739ff5fa0cc09a89d6cc483378f4691fd2feed1f4bbeaf9 Jul 21 14:42:47.126163 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:47.126144 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:42:47.828011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:47.827963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" event={"ID":"302bf22c-580e-4cbb-8dff-542048009696","Type":"ContainerStarted","Data":"8f4bc2ad7219bf7e4e986379347816a11f52f69b010822c65dd869a17a8f3b71"} Jul 21 14:42:47.828430 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:47.828068 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" event={"ID":"302bf22c-580e-4cbb-8dff-542048009696","Type":"ContainerStarted","Data":"11536d7917abd7187739ff5fa0cc09a89d6cc483378f4691fd2feed1f4bbeaf9"} Jul 21 14:42:47.846096 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:47.846040 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" podStartSLOduration=1.64370174 podStartE2EDuration="1.846022725s" totalImagesPullingTime="202.320985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:42:47.845026297 +0000 UTC m=+612.333217514" watchObservedRunningTime="2026-07-21 14:42:47.846022725 +0000 UTC m=+612.334213928" Jul 21 14:42:56.986506 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:56.986478 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:56.986919 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:56.986540 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:56.999541 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:56.999516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:42:57.884849 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:42:57.884768 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb" Jul 21 14:43:01.606406 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.606366 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d"] Jul 21 14:43:01.609875 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.609851 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.612782 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.612753 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 21 14:43:01.640432 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.640402 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d"] Jul 21 14:43:01.665600 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.665560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:01.765128 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.765090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.765128 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.765127 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.765374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.765148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.765374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.765304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.765374 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.765357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21a0d36-a578-4628-af44-6714884a76ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.765494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.765382 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9x4\" (UniqueName: \"kubernetes.io/projected/a21a0d36-a578-4628-af44-6714884a76ce-kube-api-access-sl9x4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866176 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866073 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866176 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21a0d36-a578-4628-af44-6714884a76ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866176 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9x4\" (UniqueName: \"kubernetes.io/projected/a21a0d36-a578-4628-af44-6714884a76ce-kube-api-access-sl9x4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866484 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866484 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866484 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866642 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866550 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866702 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.866803 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.866779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.868552 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.868523 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21a0d36-a578-4628-af44-6714884a76ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.868756 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.868735 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21a0d36-a578-4628-af44-6714884a76ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.875025 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.874999 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9x4\" (UniqueName: \"kubernetes.io/projected/a21a0d36-a578-4628-af44-6714884a76ce-kube-api-access-sl9x4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vdn7d\" (UID: \"a21a0d36-a578-4628-af44-6714884a76ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:01.920955 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:01.920917 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:02.067814 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:02.067786 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d"] Jul 21 14:43:02.070175 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:43:02.070144 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21a0d36_a578_4628_af44_6714884a76ce.slice/crio-6abaa5ce1e86f052ec0aaa7fe48c2e21f2f20fd65e5f777febf5d59c40b819de WatchSource:0}: Error finding container 6abaa5ce1e86f052ec0aaa7fe48c2e21f2f20fd65e5f777febf5d59c40b819de: Status 404 returned error can't find the container with id 6abaa5ce1e86f052ec0aaa7fe48c2e21f2f20fd65e5f777febf5d59c40b819de Jul 21 14:43:02.899953 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:02.899913 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" event={"ID":"a21a0d36-a578-4628-af44-6714884a76ce","Type":"ContainerStarted","Data":"09556118aead1be75335f55d7fa05d329daddcf322246ee6b012fc44e4a820e4"} Jul 21 14:43:02.900341 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:02.899965 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" event={"ID":"a21a0d36-a578-4628-af44-6714884a76ce","Type":"ContainerStarted","Data":"6abaa5ce1e86f052ec0aaa7fe48c2e21f2f20fd65e5f777febf5d59c40b819de"} Jul 21 14:43:02.924329 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:02.924276 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" podStartSLOduration=1.648484545 podStartE2EDuration="1.9242604s" totalImagesPullingTime="275.775855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:43:02.92268814 +0000 UTC m=+627.410879347" watchObservedRunningTime="2026-07-21 14:43:02.9242604 +0000 UTC m=+627.412451603" Jul 21 14:43:05.443834 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:05.443797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:09.670479 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.670447 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2"] Jul 21 14:43:09.675668 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.675646 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.678338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.678320 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 14:43:09.685030 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.685004 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2"] Jul 21 14:43:09.785100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.785073 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:09.843652 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.843616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.843820 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.843658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r277\" (UniqueName: \"kubernetes.io/projected/153a5b47-c9c5-46c7-980d-18bdb5b264c5-kube-api-access-5r277\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.843820 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.843708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.843820 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.843731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153a5b47-c9c5-46c7-980d-18bdb5b264c5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.843820 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.843753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.843820 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.843789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945133 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945171 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r277\" (UniqueName: \"kubernetes.io/projected/153a5b47-c9c5-46c7-980d-18bdb5b264c5-kube-api-access-5r277\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945188 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945237 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945213 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153a5b47-c9c5-46c7-980d-18bdb5b264c5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945583 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945265 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945583 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945283 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945701 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945694 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.945755 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.945750 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.947546 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.947522 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153a5b47-c9c5-46c7-980d-18bdb5b264c5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.947648 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.947615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153a5b47-c9c5-46c7-980d-18bdb5b264c5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.953604 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.953586 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r277\" (UniqueName: \"kubernetes.io/projected/153a5b47-c9c5-46c7-980d-18bdb5b264c5-kube-api-access-5r277\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2\" (UID: \"153a5b47-c9c5-46c7-980d-18bdb5b264c5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:09.986251 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:09.986210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:10.120527 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:10.120502 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2"] Jul 21 14:43:10.122730 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:43:10.122706 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod153a5b47_c9c5_46c7_980d_18bdb5b264c5.slice/crio-984851665e5f5ee8e0eb7c580e36151b6151d80dcaacb9e0b45735550df3a28d WatchSource:0}: Error finding container 984851665e5f5ee8e0eb7c580e36151b6151d80dcaacb9e0b45735550df3a28d: Status 404 returned error can't find the container with id 984851665e5f5ee8e0eb7c580e36151b6151d80dcaacb9e0b45735550df3a28d Jul 21 14:43:10.941958 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:10.941919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" event={"ID":"153a5b47-c9c5-46c7-980d-18bdb5b264c5","Type":"ContainerStarted","Data":"2e7d6038a2e5a0115c2b8e02d0502f402c8feb6c51fd3f21a0830d3a72b708aa"} Jul 21 14:43:10.941958 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:10.941961 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" event={"ID":"153a5b47-c9c5-46c7-980d-18bdb5b264c5","Type":"ContainerStarted","Data":"984851665e5f5ee8e0eb7c580e36151b6151d80dcaacb9e0b45735550df3a28d"} Jul 21 14:43:10.959054 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:10.959007 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" podStartSLOduration=1.765999536 podStartE2EDuration="1.958988674s" totalImagesPullingTime="192.989138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:43:10.958146598 +0000 UTC m=+635.446337803" watchObservedRunningTime="2026-07-21 14:43:10.958988674 +0000 UTC m=+635.447179860" Jul 21 14:43:11.921999 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:11.921972 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:11.922180 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:11.922045 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:11.937006 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:11.936977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:11.957494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:11.957463 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vdn7d" Jul 21 14:43:19.696393 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:19.696355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:19.762276 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:19.762204 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:19.986415 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:19.986325 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:19.986415 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:19.986375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:19.998608 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:19.998580 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:20.022361 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:20.022329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:21.005625 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:21.005595 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2" Jul 21 14:43:40.726262 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:43:40.726192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:44:10.851655 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:44:10.851607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:44:10.952975 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:44:10.952921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:44:58.886015 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:44:58.885967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:44:58.962658 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:44:58.962623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:00.048708 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.048671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:00.077804 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.077769 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b8c55c54f-k8h6r"] Jul 21 14:45:00.077975 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.077951 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" podUID="6a21e448-f731-4cad-8ece-bdd4e34bf7df" containerName="authorino" containerID="cri-o://6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b" gracePeriod=30 Jul 21 14:45:00.141563 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.141528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:00.332723 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.332697 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:45:00.405793 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.405761 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68sd9\" (UniqueName: \"kubernetes.io/projected/6a21e448-f731-4cad-8ece-bdd4e34bf7df-kube-api-access-68sd9\") pod \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " Jul 21 14:45:00.405948 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.405818 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a21e448-f731-4cad-8ece-bdd4e34bf7df-tls-cert\") pod \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\" (UID: \"6a21e448-f731-4cad-8ece-bdd4e34bf7df\") " Jul 21 14:45:00.407902 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.407868 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a21e448-f731-4cad-8ece-bdd4e34bf7df-kube-api-access-68sd9" (OuterVolumeSpecName: "kube-api-access-68sd9") pod "6a21e448-f731-4cad-8ece-bdd4e34bf7df" (UID: "6a21e448-f731-4cad-8ece-bdd4e34bf7df"). InnerVolumeSpecName "kube-api-access-68sd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:45:00.415851 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.415831 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a21e448-f731-4cad-8ece-bdd4e34bf7df-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6a21e448-f731-4cad-8ece-bdd4e34bf7df" (UID: "6a21e448-f731-4cad-8ece-bdd4e34bf7df"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:45:00.484166 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.484127 2559 generic.go:356] "Generic (PLEG): container finished" podID="6a21e448-f731-4cad-8ece-bdd4e34bf7df" containerID="6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b" exitCode=0 Jul 21 14:45:00.484166 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.484168 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" Jul 21 14:45:00.484362 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.484250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" event={"ID":"6a21e448-f731-4cad-8ece-bdd4e34bf7df","Type":"ContainerDied","Data":"6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b"} Jul 21 14:45:00.484362 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.484289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b8c55c54f-k8h6r" event={"ID":"6a21e448-f731-4cad-8ece-bdd4e34bf7df","Type":"ContainerDied","Data":"c981f779a2d51eed40dd531beed8b035707b41ddd95d3b223137cdc07cdff36a"} Jul 21 14:45:00.484362 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.484306 2559 scope.go:117] "RemoveContainer" containerID="6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b" Jul 21 14:45:00.493155 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.493136 2559 scope.go:117] "RemoveContainer" containerID="6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b" Jul 21 14:45:00.493497 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:45:00.493475 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b\": container with ID starting with 6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b not found: ID does not exist" containerID="6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b" Jul 21 14:45:00.493567 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.493507 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b"} err="failed to get container status \"6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b\": rpc error: code = NotFound desc = could not find container \"6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b\": container with ID starting with 6de29be5efe03e0a62e921e616e7f2a279a0e5db0e76bbc5a768c4b1f031cf4b not found: ID does not exist" Jul 21 14:45:00.507572 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.507552 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68sd9\" (UniqueName: \"kubernetes.io/projected/6a21e448-f731-4cad-8ece-bdd4e34bf7df-kube-api-access-68sd9\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:45:00.507572 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.507572 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a21e448-f731-4cad-8ece-bdd4e34bf7df-tls-cert\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:45:00.511601 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.511580 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b8c55c54f-k8h6r"] Jul 21 14:45:00.516964 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.516941 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b8c55c54f-k8h6r"] Jul 21 14:45:00.917699 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:45:00.917666 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a21e448-f731-4cad-8ece-bdd4e34bf7df" path="/var/lib/kubelet/pods/6a21e448-f731-4cad-8ece-bdd4e34bf7df/volumes" Jul 21 14:47:16.358137 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:47:16.358099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:17.864480 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:47:17.864442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:36.816952 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:47:36.816909 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:47:36.818117 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:47:36.818096 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:47:57.164322 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:47:57.164285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:58.328976 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:47:58.328940 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:03.135174 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:03.135132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:03.243780 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:03.243747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:03.335281 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:03.335246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:03.573702 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:03.573663 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:09.292155 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:09.292118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:09.355872 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:09.355836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:20.306236 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:20.306180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:20.403165 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:20.403123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:32.479766 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:48:32.479727 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:45.252991 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:49:45.252948 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:52:36.850147 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:52:36.850121 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:52:36.851705 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:52:36.851684 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:52:53.411276 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:52:53.411244 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:38.575773 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:53:38.575735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:44.397344 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.397309 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q"] Jul 21 14:55:44.397806 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.397725 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a21e448-f731-4cad-8ece-bdd4e34bf7df" containerName="authorino" Jul 21 14:55:44.397806 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.397737 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a21e448-f731-4cad-8ece-bdd4e34bf7df" containerName="authorino" Jul 21 14:55:44.397806 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.397802 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a21e448-f731-4cad-8ece-bdd4e34bf7df" containerName="authorino" Jul 21 14:55:44.400885 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.400863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.405490 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.405468 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3d46f128-kserve-self-signed-certs\"" Jul 21 14:55:44.426145 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.426115 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q"] Jul 21 14:55:44.497307 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.497264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tls-certs\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.497514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.497320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gx9c\" (UniqueName: \"kubernetes.io/projected/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-kube-api-access-9gx9c\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.497514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.497397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-dshm\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.497514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.497425 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-model-cache\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.497514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.497445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tmp-dir\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.497514 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.497479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-home\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.500356 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.500333 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:44.598456 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598425 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tls-certs\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.598456 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gx9c\" (UniqueName: \"kubernetes.io/projected/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-kube-api-access-9gx9c\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.598711 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598488 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-dshm\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.598711 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598505 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-model-cache\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.598711 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tmp-dir\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.598711 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-home\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.598962 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tmp-dir\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.599011 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.598947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-model-cache\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.599056 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.599023 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-home\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.600865 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.600834 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-dshm\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.601040 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.601026 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tls-certs\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.610999 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.610978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gx9c\" (UniqueName: \"kubernetes.io/projected/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-kube-api-access-9gx9c\") pod \"e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.710153 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.710056 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:44.846791 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:44.846760 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q"] Jul 21 14:55:44.848872 ip-10-0-130-112 kubenswrapper[2559]: W0721 14:55:44.848844 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0dfc04b_3a85_43c9_9ad9_b5ddc7ce59ac.slice/crio-f70691fdb4ccb1906415fc39f3f73cd853dd359c9cbd0c65404efb88b310a078 WatchSource:0}: Error finding container f70691fdb4ccb1906415fc39f3f73cd853dd359c9cbd0c65404efb88b310a078: Status 404 returned error can't find the container with id f70691fdb4ccb1906415fc39f3f73cd853dd359c9cbd0c65404efb88b310a078 Jul 21 14:55:45.602282 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:45.602241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" event={"ID":"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac","Type":"ContainerStarted","Data":"be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313"} Jul 21 14:55:45.602282 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:45.602286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" event={"ID":"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac","Type":"ContainerStarted","Data":"f70691fdb4ccb1906415fc39f3f73cd853dd359c9cbd0c65404efb88b310a078"} Jul 21 14:55:45.618558 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:45.618507 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" podStartSLOduration=1.618490419 podStartE2EDuration="1.618490419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:55:45.617889507 +0000 UTC m=+1390.106080724" watchObservedRunningTime="2026-07-21 14:55:45.618490419 +0000 UTC m=+1390.106681622" Jul 21 14:55:53.620671 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:53.620628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:54.710846 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:54.710735 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:54.710846 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:54.710814 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:54.723356 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:54.723330 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:55:55.661226 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:55:55.661196 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:56:39.868304 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:39.868270 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q"] Jul 21 14:56:39.896397 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:39.896366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:40.611611 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:40.611575 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:54.890598 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:54.890532 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" podUID="a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" containerName="main" containerID="cri-o://be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313" gracePeriod=45 Jul 21 14:56:55.133248 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.133200 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:56:55.281846 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.281821 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gx9c\" (UniqueName: \"kubernetes.io/projected/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-kube-api-access-9gx9c\") pod \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " Jul 21 14:56:55.282030 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.281870 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-model-cache\") pod \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " Jul 21 14:56:55.282030 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.281897 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tls-certs\") pod \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " Jul 21 14:56:55.282030 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.281933 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-dshm\") pod \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " Jul 21 14:56:55.282030 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.281950 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-home\") pod \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " Jul 21 14:56:55.282030 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.282023 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tmp-dir\") pod \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\" (UID: \"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac\") " Jul 21 14:56:55.282338 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.282154 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" (UID: "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:55.282397 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.282326 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-home" (OuterVolumeSpecName: "home") pod "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" (UID: "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:55.282397 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.282347 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" (UID: "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:55.282397 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.282364 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:56:55.284146 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.284121 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" (UID: "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:56:55.284263 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.284156 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-dshm" (OuterVolumeSpecName: "dshm") pod "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" (UID: "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:55.284263 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.284176 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-kube-api-access-9gx9c" (OuterVolumeSpecName: "kube-api-access-9gx9c") pod "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" (UID: "a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac"). InnerVolumeSpecName "kube-api-access-9gx9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:56:55.382877 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.382842 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:56:55.382877 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.382871 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gx9c\" (UniqueName: \"kubernetes.io/projected/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-kube-api-access-9gx9c\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:56:55.382877 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.382881 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:56:55.383100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.382890 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:56:55.383100 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.382898 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 14:56:55.946972 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.946940 2559 generic.go:356] "Generic (PLEG): container finished" podID="a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" containerID="be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313" exitCode=0 Jul 21 14:56:55.946972 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.946976 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" Jul 21 14:56:55.947494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.947008 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" event={"ID":"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac","Type":"ContainerDied","Data":"be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313"} Jul 21 14:56:55.947494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.947037 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q" event={"ID":"a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac","Type":"ContainerDied","Data":"f70691fdb4ccb1906415fc39f3f73cd853dd359c9cbd0c65404efb88b310a078"} Jul 21 14:56:55.947494 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.947053 2559 scope.go:117] "RemoveContainer" containerID="be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313" Jul 21 14:56:55.956154 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.956136 2559 scope.go:117] "RemoveContainer" containerID="be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313" Jul 21 14:56:55.956423 ip-10-0-130-112 kubenswrapper[2559]: E0721 14:56:55.956402 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313\": container with ID starting with be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313 not found: ID does not exist" containerID="be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313" Jul 21 14:56:55.956482 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.956431 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313"} err="failed to get container status \"be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313\": rpc error: code = NotFound desc = could not find container \"be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313\": container with ID starting with be259844cf2189ba11fb34de804ebc9467d812414243573170d303e226bcc313 not found: ID does not exist" Jul 21 14:56:55.968339 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.968313 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q"] Jul 21 14:56:55.974664 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:55.974643 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3d46f128-kserve-75997cc94c-tc87q"] Jul 21 14:56:56.916923 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:56:56.916892 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" path="/var/lib/kubelet/pods/a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac/volumes" Jul 21 14:57:36.879325 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:57:36.879298 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:57:36.882607 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:57:36.882585 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 14:57:54.463085 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:57:54.463048 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:54.564842 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:57:54.564792 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:10.795702 ip-10-0-130-112 kubenswrapper[2559]: I0721 14:58:10.795609 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:03.386055 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:03.386023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:17.496693 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:17.496650 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:25.773854 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:25.773812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:32.579855 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:32.579811 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:35.627079 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:35.627040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:35.718097 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:35.718062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:36.022746 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:36.022651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:41.899073 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:41.899031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:41.999081 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:41.999046 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:42.120957 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:42.120923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:42.431273 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:42.431190 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:43.015014 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:43.014978 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:44.512364 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:44.512327 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:44.637853 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:44.637807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:44.801909 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:44.801874 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:47.065877 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:47.065842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:47.167664 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:00:47.167631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:00.553577 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:00.553496 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:00.653798 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:00.653767 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:20.565821 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:20.565789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:25.164838 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:25.164802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:25.316007 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:25.315973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:26.233201 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:26.233158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:26.310983 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:26.310951 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:26.373377 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:26.373341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:26.893420 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:26.893389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:28.890010 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:28.889972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:29.086929 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:29.086902 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:29.194454 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:29.194338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:29.550651 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:29.550620 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:29.651482 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:01:29.651433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:04.449299 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:04.449258 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:08.966709 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:08.966672 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:09.132199 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:09.132163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:09.373571 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:09.373537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:10.049050 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:10.049005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:10.145332 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:10.145296 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:10.465552 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:10.465458 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:12.072516 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:12.072484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:12.164707 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:12.164674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:12.236629 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:12.236597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:12.340904 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:12.340798 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:12.935199 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:12.935167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:13.042333 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:13.042301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:36.909881 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:36.909843 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:02:36.915449 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:36.915427 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:02:38.883855 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:38.883818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:38.985537 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:38.985503 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:39.765072 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:39.765032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:39.865866 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:39.865832 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:48.513605 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:48.513570 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:53.024677 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:53.024640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:54.246271 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:54.246204 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:54.484492 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:54.484445 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:54.710837 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:54.710799 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:55.013684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:55.013590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:55.642326 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:55.642294 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:57.351332 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:57.351297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:57.668792 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:57.668695 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:58.060661 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:58.060619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:58.201991 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:58.201958 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:58.296204 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:58.296157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:58.396497 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:02:58.396396 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:32.657916 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:32.657870 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:37.303018 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:37.302975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:37.663272 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:37.663159 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:37.723911 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:37.723871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:38.181684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:38.181641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:38.285029 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:38.284978 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:38.381969 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:38.381934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:39.780570 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:39.780529 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:45.495885 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:45.495848 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:45.594348 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:45.594303 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:45.692801 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:45.692763 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:45.794185 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:45.794143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:46.136436 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:03:46.136337 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:12.111716 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:12.111679 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:18.431598 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:18.431556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:18.599094 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:18.599056 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:18.699438 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:18.699341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:34.570566 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:34.570529 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:40.350280 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:40.350234 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:40.404544 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:40.404505 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:40.511717 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:40.511684 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:40.611132 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:40.611028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:51.107952 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:51.107892 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:51.916605 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:51.916568 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:58.728283 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:58.728208 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:58.834502 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:58.834459 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:58.928849 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:58.928808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:59.028846 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:04:59.028749 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:04.903818 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:04.903782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:05.001873 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:05.001829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:05.101895 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:05.101859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:06.683083 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:06.683047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:12.913716 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:12.913673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:15.221420 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:15.221382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:19.314737 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:19.314691 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:19.544559 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:19.544522 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:19.641932 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:19.641843 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:19.918155 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:19.918029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:20.241412 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:20.241326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:22.882174 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:22.882132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:23.516465 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:23.516423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:23.615669 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:23.615639 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:23.764758 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:23.764707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:23.865371 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:23.865328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:27.319128 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:27.319089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:27.947738 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:27.947701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:28.527596 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:28.527560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:29.092106 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:29.092071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:34.106814 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:34.106781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:34.212744 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:34.212710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:45.160146 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:45.160100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:49.627018 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:49.626979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:50.043148 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:50.043118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:56.179536 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:56.179489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:56.293290 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:56.293250 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:56.745581 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:56.745541 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:57.010288 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:57.010195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:57.167406 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:05:57.167376 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:03.048129 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:03.048090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:03.117521 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:03.117479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:03.209131 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:03.209094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:09.139037 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:09.138996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:09.215177 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:09.215140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:15.937460 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:15.937423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:16.312369 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:16.312320 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:16.559882 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:16.559849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:18.993540 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:18.993504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:19.091370 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:19.091339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:19.162381 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:19.162340 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:19.338787 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:19.338752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:19.759706 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:19.759621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:19.859353 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:19.859317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:23.477044 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:23.477003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:27.150140 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.150101 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2"] Jul 21 15:06:27.150765 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.150743 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" containerName="main" Jul 21 15:06:27.150854 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.150768 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" containerName="main" Jul 21 15:06:27.150921 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.150907 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0dfc04b-3a85-43c9-9ad9-b5ddc7ce59ac" containerName="main" Jul 21 15:06:27.153995 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.153974 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.159098 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.159074 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e94d052\"/\"kube-root-ca.crt\"" Jul 21 15:06:27.159196 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.159074 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e94d052\"/\"openshift-service-ca.crt\"" Jul 21 15:06:27.160204 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.160183 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e94d052\"/\"e2e-lifecycle-model-5e94d052-kserve-self-signed-certs\"" Jul 21 15:06:27.160322 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.160275 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e94d052\"/\"default-dockercfg-9wr4w\"" Jul 21 15:06:27.170934 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.170907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2"] Jul 21 15:06:27.183166 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.183141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tmp-dir\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.183309 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.183172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tls-certs\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.183309 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.183204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-home\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.183309 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.183256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-dshm\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.183309 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.183277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-model-cache\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.183451 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.183350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzv8b\" (UniqueName: \"kubernetes.io/projected/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-kube-api-access-wzv8b\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.225281 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.225227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:27.284440 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tmp-dir\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.284440 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284445 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tls-certs\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.284696 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-home\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.284696 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-dshm\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.284696 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284557 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-model-cache\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.284696 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv8b\" (UniqueName: \"kubernetes.io/projected/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-kube-api-access-wzv8b\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.284925 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tmp-dir\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.284993 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.284936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-home\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.285048 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.285019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-model-cache\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.286803 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.286781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-dshm\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.287230 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.287193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tls-certs\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.298631 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.298605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv8b\" (UniqueName: \"kubernetes.io/projected/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-kube-api-access-wzv8b\") pod \"e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.463427 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.463334 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:27.598746 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.598718 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2"] Jul 21 15:06:27.600594 ip-10-0-130-112 kubenswrapper[2559]: W0721 15:06:27.600567 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b64e1ab_9b5f_4c05_a376_23e0227bbf19.slice/crio-1d860fb58af18773a648035030a1bf1a6ea64720b0a5804e1f5415a616e5451b WatchSource:0}: Error finding container 1d860fb58af18773a648035030a1bf1a6ea64720b0a5804e1f5415a616e5451b: Status 404 returned error can't find the container with id 1d860fb58af18773a648035030a1bf1a6ea64720b0a5804e1f5415a616e5451b Jul 21 15:06:27.728084 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.727997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" event={"ID":"4b64e1ab-9b5f-4c05-a376-23e0227bbf19","Type":"ContainerStarted","Data":"ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80"} Jul 21 15:06:27.728084 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.728040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" event={"ID":"4b64e1ab-9b5f-4c05-a376-23e0227bbf19","Type":"ContainerStarted","Data":"1d860fb58af18773a648035030a1bf1a6ea64720b0a5804e1f5415a616e5451b"} Jul 21 15:06:27.752355 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:27.752301 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" podStartSLOduration=0.752283634 podStartE2EDuration="752.283634ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:06:27.750554999 +0000 UTC m=+2032.238746224" watchObservedRunningTime="2026-07-21 15:06:27.752283634 +0000 UTC m=+2032.240474838" Jul 21 15:06:37.463893 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:37.463859 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:37.464406 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:37.463909 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:37.476400 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:37.476376 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:37.792512 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:37.792483 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:06:40.967726 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:40.967684 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:06:41.026056 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:06:41.026023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:08.738037 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:08.737943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:11.223449 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:11.223415 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:11.787595 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:11.787561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:14.045327 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:14.045292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:17.699157 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:17.699116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:17.755777 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:17.755746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:18.967863 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:18.967830 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2"] Jul 21 15:07:19.026414 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:19.026367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:33.988589 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:33.988547 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" podUID="4b64e1ab-9b5f-4c05-a376-23e0227bbf19" containerName="main" containerID="cri-o://ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80" gracePeriod=45 Jul 21 15:07:34.238550 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.238516 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:07:34.245974 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.245916 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tmp-dir\") pod \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " Jul 21 15:07:34.245974 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.245956 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tls-certs\") pod \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " Jul 21 15:07:34.246173 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.245981 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-model-cache\") pod \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " Jul 21 15:07:34.246173 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzv8b\" (UniqueName: \"kubernetes.io/projected/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-kube-api-access-wzv8b\") pod \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " Jul 21 15:07:34.246173 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246090 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-dshm\") pod \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " Jul 21 15:07:34.246173 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246149 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-home\") pod \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\" (UID: \"4b64e1ab-9b5f-4c05-a376-23e0227bbf19\") " Jul 21 15:07:34.246396 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246209 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b64e1ab-9b5f-4c05-a376-23e0227bbf19" (UID: "4b64e1ab-9b5f-4c05-a376-23e0227bbf19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:07:34.246396 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246248 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b64e1ab-9b5f-4c05-a376-23e0227bbf19" (UID: "4b64e1ab-9b5f-4c05-a376-23e0227bbf19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:07:34.246482 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246403 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-home" (OuterVolumeSpecName: "home") pod "4b64e1ab-9b5f-4c05-a376-23e0227bbf19" (UID: "4b64e1ab-9b5f-4c05-a376-23e0227bbf19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:07:34.246523 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246498 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:07:34.246523 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246512 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:07:34.246523 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.246522 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:07:34.248173 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.248150 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-kube-api-access-wzv8b" (OuterVolumeSpecName: "kube-api-access-wzv8b") pod "4b64e1ab-9b5f-4c05-a376-23e0227bbf19" (UID: "4b64e1ab-9b5f-4c05-a376-23e0227bbf19"). InnerVolumeSpecName "kube-api-access-wzv8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:07:34.248495 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.248461 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-dshm" (OuterVolumeSpecName: "dshm") pod "4b64e1ab-9b5f-4c05-a376-23e0227bbf19" (UID: "4b64e1ab-9b5f-4c05-a376-23e0227bbf19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:07:34.248597 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.248509 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b64e1ab-9b5f-4c05-a376-23e0227bbf19" (UID: "4b64e1ab-9b5f-4c05-a376-23e0227bbf19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:07:34.347588 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.347551 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:07:34.347588 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.347582 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzv8b\" (UniqueName: \"kubernetes.io/projected/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-kube-api-access-wzv8b\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:07:34.347588 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:34.347592 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b64e1ab-9b5f-4c05-a376-23e0227bbf19-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:07:35.058717 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.058681 2559 generic.go:356] "Generic (PLEG): container finished" podID="4b64e1ab-9b5f-4c05-a376-23e0227bbf19" containerID="ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80" exitCode=0 Jul 21 15:07:35.059135 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.058732 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" Jul 21 15:07:35.059135 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.058756 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" event={"ID":"4b64e1ab-9b5f-4c05-a376-23e0227bbf19","Type":"ContainerDied","Data":"ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80"} Jul 21 15:07:35.059135 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.058797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2" event={"ID":"4b64e1ab-9b5f-4c05-a376-23e0227bbf19","Type":"ContainerDied","Data":"1d860fb58af18773a648035030a1bf1a6ea64720b0a5804e1f5415a616e5451b"} Jul 21 15:07:35.059135 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.058814 2559 scope.go:117] "RemoveContainer" containerID="ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80" Jul 21 15:07:35.067418 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.067400 2559 scope.go:117] "RemoveContainer" containerID="ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80" Jul 21 15:07:35.067673 ip-10-0-130-112 kubenswrapper[2559]: E0721 15:07:35.067654 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80\": container with ID starting with ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80 not found: ID does not exist" containerID="ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80" Jul 21 15:07:35.067744 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.067681 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80"} err="failed to get container status \"ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80\": rpc error: code = NotFound desc = could not find container \"ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80\": container with ID starting with ced1a1492c8926de140478895945e74e0f2a7c50da2bb687b74c8d25f0721f80 not found: ID does not exist" Jul 21 15:07:35.075834 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.075804 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2"] Jul 21 15:07:35.079834 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:35.079815 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5e94d052/e2e-lifecycle-model-5e94d052-kserve-55cc64dcd7-gthm2"] Jul 21 15:07:36.916690 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:36.916657 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b64e1ab-9b5f-4c05-a376-23e0227bbf19" path="/var/lib/kubelet/pods/4b64e1ab-9b5f-4c05-a376-23e0227bbf19/volumes" Jul 21 15:07:36.941791 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:36.941767 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:07:36.946992 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:36.946971 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:07:39.698661 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:39.698624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:39.796523 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:39.796485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:55.108920 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:55.108867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:07:56.204448 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:07:56.204404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:01.823684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:01.823648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:01.891037 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:01.890998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:02.103576 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:02.103496 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:02.333185 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:02.333148 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:03.375823 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:03.375787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:03.476663 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:03.476633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:03.576547 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:03.576512 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:04.371457 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:04.371419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:04.473055 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:04.473018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:04.571160 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:04.571121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:05.883587 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:05.883534 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:11.625655 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:11.625615 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:11.703037 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:11.703000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:12.359962 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:12.359920 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:18.179129 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:18.179088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:18.288235 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:18.288178 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:18.388535 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:18.388502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:29.006706 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:29.006595 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:29.294642 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:29.294597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:50.267619 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:50.267579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:56.924472 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:56.924428 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:57.024920 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:57.024882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:57.283059 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:57.283027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:57.854526 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:57.854487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:08:57.979478 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:08:57.979419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:00.965040 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:00.964999 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:01.062059 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:01.062020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:01.161822 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:01.161770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:01.283732 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:01.283694 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:01.639050 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:01.638949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:01.748934 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:01.748884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:06.038697 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:06.038659 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:07.863103 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:07.863064 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:07.964545 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:07.964511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:08.995842 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:08.995796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:09.088681 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:09.088625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:09.258489 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:09.258389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:09.529409 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:09.529312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:09.765193 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:09.765153 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:09.865014 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:09.864979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:13.487098 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:13.487058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:22.523901 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:22.523861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:22.626241 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:22.626183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:24.483322 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.483281 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg"] Jul 21 15:09:24.483955 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.483915 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b64e1ab-9b5f-4c05-a376-23e0227bbf19" containerName="main" Jul 21 15:09:24.483955 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.483941 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b64e1ab-9b5f-4c05-a376-23e0227bbf19" containerName="main" Jul 21 15:09:24.484143 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.484071 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b64e1ab-9b5f-4c05-a376-23e0227bbf19" containerName="main" Jul 21 15:09:24.487585 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.487567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.490175 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.490153 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d6d63a\"/\"kube-root-ca.crt\"" Jul 21 15:09:24.491391 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.491371 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d6d63a\"/\"default-dockercfg-qbt5h\"" Jul 21 15:09:24.491502 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.491393 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d6d63a\"/\"test-model-d6d63a-kserve-self-signed-certs\"" Jul 21 15:09:24.491502 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.491412 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d6d63a\"/\"openshift-service-ca.crt\"" Jul 21 15:09:24.495285 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.495264 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg"] Jul 21 15:09:24.588117 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.588071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca02f692-0b63-400e-aac4-28ebcd7585df-tls-certs\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.588330 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.588200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-home\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.588330 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.588272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-tmp-dir\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.588330 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.588291 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-dshm\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.588330 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.588312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-model-cache\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.588532 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.588390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc749\" (UniqueName: \"kubernetes.io/projected/ca02f692-0b63-400e-aac4-28ebcd7585df-kube-api-access-lc749\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.589734 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.589709 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:24.689837 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.689796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca02f692-0b63-400e-aac4-28ebcd7585df-tls-certs\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690040 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.689892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-home\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690040 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.689938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-tmp-dir\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690040 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.689958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-dshm\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690040 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.689984 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-model-cache\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690040 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.690037 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc749\" (UniqueName: \"kubernetes.io/projected/ca02f692-0b63-400e-aac4-28ebcd7585df-kube-api-access-lc749\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690471 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.690447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-tmp-dir\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690541 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.690474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-home\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.690541 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.690503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-model-cache\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.692263 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.692237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-dshm\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.692470 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.692451 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca02f692-0b63-400e-aac4-28ebcd7585df-tls-certs\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.698337 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.698316 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc749\" (UniqueName: \"kubernetes.io/projected/ca02f692-0b63-400e-aac4-28ebcd7585df-kube-api-access-lc749\") pod \"test-model-d6d63a-kserve-6fbddd9bd5-96czg\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.797384 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.797349 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:24.935640 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:24.935618 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg"] Jul 21 15:09:24.938125 ip-10-0-130-112 kubenswrapper[2559]: W0721 15:09:24.938099 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca02f692_0b63_400e_aac4_28ebcd7585df.slice/crio-f8f3f0d4435c30e0d6a6ee2f8b51375b0cf338092c349ecd43d0734abddfb16b WatchSource:0}: Error finding container f8f3f0d4435c30e0d6a6ee2f8b51375b0cf338092c349ecd43d0734abddfb16b: Status 404 returned error can't find the container with id f8f3f0d4435c30e0d6a6ee2f8b51375b0cf338092c349ecd43d0734abddfb16b Jul 21 15:09:25.604067 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:25.604032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" event={"ID":"ca02f692-0b63-400e-aac4-28ebcd7585df","Type":"ContainerStarted","Data":"a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc"} Jul 21 15:09:25.604067 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:25.604073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" event={"ID":"ca02f692-0b63-400e-aac4-28ebcd7585df","Type":"ContainerStarted","Data":"f8f3f0d4435c30e0d6a6ee2f8b51375b0cf338092c349ecd43d0734abddfb16b"} Jul 21 15:09:25.622543 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:25.622489 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" podStartSLOduration=1.6224748679999998 podStartE2EDuration="1.622474868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:09:25.62061635 +0000 UTC m=+2210.108807579" watchObservedRunningTime="2026-07-21 15:09:25.622474868 +0000 UTC m=+2210.110666072" Jul 21 15:09:30.796896 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:30.796846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:32.549239 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:32.549172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:09:34.797900 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:34.797855 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:34.798324 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:34.797921 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:34.810699 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:34.810675 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:35.664051 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:35.664022 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:09:55.592937 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:09:55.592892 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:10:05.650422 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:05.650388 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:10:38.335838 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:38.335802 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg"] Jul 21 15:10:38.357742 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:38.357703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:10:42.360685 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:42.360646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:10:49.932747 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:49.932709 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:10:53.356356 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.356314 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" podUID="ca02f692-0b63-400e-aac4-28ebcd7585df" containerName="main" containerID="cri-o://a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc" gracePeriod=45 Jul 21 15:10:53.608925 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.608846 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:10:53.663899 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.663867 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-dshm\") pod \"ca02f692-0b63-400e-aac4-28ebcd7585df\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " Jul 21 15:10:53.664063 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.663942 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca02f692-0b63-400e-aac4-28ebcd7585df-tls-certs\") pod \"ca02f692-0b63-400e-aac4-28ebcd7585df\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " Jul 21 15:10:53.664063 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.663972 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc749\" (UniqueName: \"kubernetes.io/projected/ca02f692-0b63-400e-aac4-28ebcd7585df-kube-api-access-lc749\") pod \"ca02f692-0b63-400e-aac4-28ebcd7585df\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " Jul 21 15:10:53.664063 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.664020 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-home\") pod \"ca02f692-0b63-400e-aac4-28ebcd7585df\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " Jul 21 15:10:53.664063 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.664055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-tmp-dir\") pod \"ca02f692-0b63-400e-aac4-28ebcd7585df\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " Jul 21 15:10:53.664273 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.664119 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-model-cache\") pod \"ca02f692-0b63-400e-aac4-28ebcd7585df\" (UID: \"ca02f692-0b63-400e-aac4-28ebcd7585df\") " Jul 21 15:10:53.664620 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.664325 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-home" (OuterVolumeSpecName: "home") pod "ca02f692-0b63-400e-aac4-28ebcd7585df" (UID: "ca02f692-0b63-400e-aac4-28ebcd7585df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:10:53.664620 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.664417 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca02f692-0b63-400e-aac4-28ebcd7585df" (UID: "ca02f692-0b63-400e-aac4-28ebcd7585df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:10:53.664620 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.664496 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:10:53.664620 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.664538 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca02f692-0b63-400e-aac4-28ebcd7585df" (UID: "ca02f692-0b63-400e-aac4-28ebcd7585df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:10:53.666266 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.666246 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-dshm" (OuterVolumeSpecName: "dshm") pod "ca02f692-0b63-400e-aac4-28ebcd7585df" (UID: "ca02f692-0b63-400e-aac4-28ebcd7585df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:10:53.666567 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.666543 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca02f692-0b63-400e-aac4-28ebcd7585df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca02f692-0b63-400e-aac4-28ebcd7585df" (UID: "ca02f692-0b63-400e-aac4-28ebcd7585df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:10:53.666567 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.666554 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca02f692-0b63-400e-aac4-28ebcd7585df-kube-api-access-lc749" (OuterVolumeSpecName: "kube-api-access-lc749") pod "ca02f692-0b63-400e-aac4-28ebcd7585df" (UID: "ca02f692-0b63-400e-aac4-28ebcd7585df"). InnerVolumeSpecName "kube-api-access-lc749". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:10:53.765774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.765733 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:10:53.765774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.765767 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:10:53.765774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.765775 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca02f692-0b63-400e-aac4-28ebcd7585df-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:10:53.765774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.765784 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca02f692-0b63-400e-aac4-28ebcd7585df-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:10:53.766064 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:53.765792 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc749\" (UniqueName: \"kubernetes.io/projected/ca02f692-0b63-400e-aac4-28ebcd7585df-kube-api-access-lc749\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:10:54.034973 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.034934 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca02f692-0b63-400e-aac4-28ebcd7585df" containerID="a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc" exitCode=0 Jul 21 15:10:54.035131 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.034978 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" Jul 21 15:10:54.035131 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.035016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" event={"ID":"ca02f692-0b63-400e-aac4-28ebcd7585df","Type":"ContainerDied","Data":"a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc"} Jul 21 15:10:54.035131 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.035058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg" event={"ID":"ca02f692-0b63-400e-aac4-28ebcd7585df","Type":"ContainerDied","Data":"f8f3f0d4435c30e0d6a6ee2f8b51375b0cf338092c349ecd43d0734abddfb16b"} Jul 21 15:10:54.035131 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.035076 2559 scope.go:117] "RemoveContainer" containerID="a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc" Jul 21 15:10:54.046482 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.046463 2559 scope.go:117] "RemoveContainer" containerID="a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc" Jul 21 15:10:54.046734 ip-10-0-130-112 kubenswrapper[2559]: E0721 15:10:54.046712 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc\": container with ID starting with a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc not found: ID does not exist" containerID="a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc" Jul 21 15:10:54.046830 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.046739 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc"} err="failed to get container status \"a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc\": rpc error: code = NotFound desc = could not find container \"a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc\": container with ID starting with a0f7133f6b2d2e492cc4a81d15e686e74ba8bdf71e78a9cdeeadc69663b591bc not found: ID does not exist" Jul 21 15:10:54.057847 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.057822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg"] Jul 21 15:10:54.063384 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.063354 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d6d63a/test-model-d6d63a-kserve-6fbddd9bd5-96czg"] Jul 21 15:10:54.504592 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.504495 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:10:54.901681 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.901648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:10:54.916572 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:10:54.916544 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca02f692-0b63-400e-aac4-28ebcd7585df" path="/var/lib/kubelet/pods/ca02f692-0b63-400e-aac4-28ebcd7585df/volumes" Jul 21 15:11:00.657389 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:00.657346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:00.727983 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:00.727945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:00.954394 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:00.954299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:01.178415 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:01.178382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:01.826721 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:01.826684 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:04.616355 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:04.616317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:11.727127 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:11.727089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:16.368969 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:16.368931 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:16.789080 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:16.789048 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:22.671165 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:22.671128 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:22.772408 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:22.772371 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:23.006924 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:23.006826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:23.155621 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:23.155586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:23.485327 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:23.485290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:23.805894 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:23.805860 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:24.070567 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:24.070472 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:26.812061 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:26.812024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:26.893664 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:26.893622 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:26.972749 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:26.972704 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:27.073225 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:27.073119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:27.589571 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:27.589479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:30.203958 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:30.203835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:31.313077 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:31.313031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:31.787275 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:31.787209 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:32.096787 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:32.096669 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:32.408288 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:32.408157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:34.846970 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:34.846922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:34.917956 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:34.917925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:35.141924 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:35.141837 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:35.244170 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:35.244133 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:35.625030 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:35.624996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:35.725349 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:35.725313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:39.357449 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:39.357416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:47.895976 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:47.895931 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:48.000238 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:48.000186 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:56.498204 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:56.498169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:56.600598 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:56.600540 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:58.059907 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:58.059867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:11:58.159955 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:11:58.159919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:03.322784 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:03.322745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:07.464485 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:07.464452 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:07.863498 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:07.863460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:13.768958 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:13.768921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:13.868262 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:13.868206 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:14.034189 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:14.034156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:14.305065 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:14.304951 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:14.956061 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:14.956006 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:15.091209 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:15.091169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:23.975974 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:23.975923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:29.974543 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:29.974498 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:34.470391 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:34.470351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:34.857466 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:34.857430 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:36.972241 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:36.972192 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:12:36.979673 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:36.979653 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:12:40.451168 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:40.451121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:40.556360 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:40.556317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:40.924625 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:40.924593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:40.987585 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:40.987544 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:41.212991 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:41.212898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:41.451441 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:41.451407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:41.784923 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:41.784887 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:42.100018 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:42.099913 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:42.458652 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:42.458549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:45.315014 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:45.314970 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:45.401852 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:45.401804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:45.467773 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:45.467740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:45.714493 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:45.714404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:45.974587 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:45.974474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:46.075314 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:46.075279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:49.815879 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:49.815842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:50.143857 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:50.143757 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:50.448882 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:50.448793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:50.812708 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:50.812667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:53.467140 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:53.467101 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:53.546390 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:53.546348 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:53.617199 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:53.617165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:53.811333 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:53.811290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:54.217529 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:54.217449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:54.320721 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:54.320676 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:12:57.950420 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:12:57.950340 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:07.583964 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:07.583930 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:07.689584 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:07.689546 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:13.919529 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.919497 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn"] Jul 21 15:13:13.919962 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.919866 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca02f692-0b63-400e-aac4-28ebcd7585df" containerName="main" Jul 21 15:13:13.919962 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.919876 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca02f692-0b63-400e-aac4-28ebcd7585df" containerName="main" Jul 21 15:13:13.919962 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.919952 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca02f692-0b63-400e-aac4-28ebcd7585df" containerName="main" Jul 21 15:13:13.922966 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.922946 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:13.927942 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.927922 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9b06ae\"/\"kube-root-ca.crt\"" Jul 21 15:13:13.928648 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.928629 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b06ae\"/\"default-dockercfg-skszr\"" Jul 21 15:13:13.928877 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.928859 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9b06ae\"/\"openshift-service-ca.crt\"" Jul 21 15:13:13.929433 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.929248 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b06ae\"/\"auth-test-model-9b06ae-kserve-self-signed-certs\"" Jul 21 15:13:13.938497 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.938478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn"] Jul 21 15:13:13.983049 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:13.983017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:14.005240 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.005180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-model-cache\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.005388 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.005247 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tmp-dir\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.005388 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.005282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-home\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.005388 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.005357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tls-certs\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.005538 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.005396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-dshm\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.005538 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.005436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbwq\" (UniqueName: \"kubernetes.io/projected/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-kube-api-access-mnbwq\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.105906 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.105879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-model-cache\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106048 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.105908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tmp-dir\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106048 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.105929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-home\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106048 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.105970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tls-certs\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106048 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.105991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-dshm\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106048 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.106013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbwq\" (UniqueName: \"kubernetes.io/projected/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-kube-api-access-mnbwq\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106356 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.106269 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-model-cache\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106412 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.106347 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tmp-dir\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.106412 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.106391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-home\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.108397 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.108374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-dshm\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.108617 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.108602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tls-certs\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.113885 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.113867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbwq\" (UniqueName: \"kubernetes.io/projected/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-kube-api-access-mnbwq\") pod \"auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.232479 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.232382 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:14.361471 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.361446 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn"] Jul 21 15:13:14.363355 ip-10-0-130-112 kubenswrapper[2559]: W0721 15:13:14.363327 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7553ab1_f268_429e_a1e5_e8e4a5c2c648.slice/crio-4da6225e1f6c41a5d5ab2aa71616aeea71e6164cc781334854d17902d7db3203 WatchSource:0}: Error finding container 4da6225e1f6c41a5d5ab2aa71616aeea71e6164cc781334854d17902d7db3203: Status 404 returned error can't find the container with id 4da6225e1f6c41a5d5ab2aa71616aeea71e6164cc781334854d17902d7db3203 Jul 21 15:13:14.715573 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.715535 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" event={"ID":"c7553ab1-f268-429e-a1e5-e8e4a5c2c648","Type":"ContainerStarted","Data":"7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b"} Jul 21 15:13:14.715775 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.715582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" event={"ID":"c7553ab1-f268-429e-a1e5-e8e4a5c2c648","Type":"ContainerStarted","Data":"4da6225e1f6c41a5d5ab2aa71616aeea71e6164cc781334854d17902d7db3203"} Jul 21 15:13:14.732002 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:14.731955 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" podStartSLOduration=1.731942574 podStartE2EDuration="1.731942574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:13:14.731558396 +0000 UTC m=+2439.219749601" watchObservedRunningTime="2026-07-21 15:13:14.731942574 +0000 UTC m=+2439.220133778" Jul 21 15:13:16.455117 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:16.455080 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:16.512010 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:16.511958 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:22.603940 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:22.603898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:24.232931 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:24.232895 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:24.233444 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:24.232948 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:24.248924 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:24.248882 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:24.774118 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:24.774091 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:13:31.783284 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:31.783245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:13:32.136245 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:13:32.136126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:14:57.766810 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:14:57.766771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:15:17.662174 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:17.662142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:15:42.751758 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:42.751712 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn"] Jul 21 15:15:42.772098 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:42.772064 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:15:46.121282 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:46.121246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:15:48.993707 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:48.993677 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7"] Jul 21 15:15:48.996984 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:48.996965 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:48.999678 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:48.999663 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b06ae\"/\"sub-test-model-9b06ae-kserve-self-signed-certs\"" Jul 21 15:15:49.006105 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.006077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7"] Jul 21 15:15:49.023912 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.023873 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:15:49.127712 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.127667 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjb4d\" (UniqueName: \"kubernetes.io/projected/930a9075-a663-402d-b0e4-8097443de3c1-kube-api-access-vjb4d\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.127712 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.127718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-home\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.128001 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.127739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a9075-a663-402d-b0e4-8097443de3c1-tls-certs\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.128001 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.127758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-tmp-dir\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.128001 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.127859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-model-cache\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.128001 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.127910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-dshm\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.228856 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.228808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjb4d\" (UniqueName: \"kubernetes.io/projected/930a9075-a663-402d-b0e4-8097443de3c1-kube-api-access-vjb4d\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.228856 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.228868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-home\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.229080 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.228891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a9075-a663-402d-b0e4-8097443de3c1-tls-certs\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.229080 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.228915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-tmp-dir\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.229080 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.228979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-model-cache\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.229080 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.229019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-dshm\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.229374 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.229349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-home\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.229374 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.229364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-tmp-dir\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.229455 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.229417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-model-cache\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.231396 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.231378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-dshm\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.231497 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.231481 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a9075-a663-402d-b0e4-8097443de3c1-tls-certs\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.236941 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.236916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjb4d\" (UniqueName: \"kubernetes.io/projected/930a9075-a663-402d-b0e4-8097443de3c1-kube-api-access-vjb4d\") pod \"sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.307644 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.307606 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:49.443932 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.443906 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7"] Jul 21 15:15:49.446177 ip-10-0-130-112 kubenswrapper[2559]: W0721 15:15:49.446149 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod930a9075_a663_402d_b0e4_8097443de3c1.slice/crio-e1d14b9f75b1ab8e1b477fc4c2e489da18767c91cc3165e0cdc5caf98ce687f2 WatchSource:0}: Error finding container e1d14b9f75b1ab8e1b477fc4c2e489da18767c91cc3165e0cdc5caf98ce687f2: Status 404 returned error can't find the container with id e1d14b9f75b1ab8e1b477fc4c2e489da18767c91cc3165e0cdc5caf98ce687f2 Jul 21 15:15:49.470631 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:49.470601 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" event={"ID":"930a9075-a663-402d-b0e4-8097443de3c1","Type":"ContainerStarted","Data":"e1d14b9f75b1ab8e1b477fc4c2e489da18767c91cc3165e0cdc5caf98ce687f2"} Jul 21 15:15:50.477749 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:50.477706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" event={"ID":"930a9075-a663-402d-b0e4-8097443de3c1","Type":"ContainerStarted","Data":"4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6"} Jul 21 15:15:50.494543 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:50.494472 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" podStartSLOduration=2.494451032 podStartE2EDuration="2.494451032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:15:50.494011846 +0000 UTC m=+2594.982203072" watchObservedRunningTime="2026-07-21 15:15:50.494451032 +0000 UTC m=+2594.982642240" Jul 21 15:15:57.773018 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:57.772907 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" podUID="c7553ab1-f268-429e-a1e5-e8e4a5c2c648" containerName="main" containerID="cri-o://7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b" gracePeriod=45 Jul 21 15:15:58.022668 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.022640 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:15:58.117465 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117414 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tmp-dir\") pod \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " Jul 21 15:15:58.117651 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117524 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-dshm\") pod \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " Jul 21 15:15:58.117651 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117583 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tls-certs\") pod \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " Jul 21 15:15:58.117651 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117605 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnbwq\" (UniqueName: \"kubernetes.io/projected/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-kube-api-access-mnbwq\") pod \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " Jul 21 15:15:58.117651 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117635 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-model-cache\") pod \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " Jul 21 15:15:58.117651 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117639 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7553ab1-f268-429e-a1e5-e8e4a5c2c648" (UID: "c7553ab1-f268-429e-a1e5-e8e4a5c2c648"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:15:58.117940 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117656 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-home\") pod \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\" (UID: \"c7553ab1-f268-429e-a1e5-e8e4a5c2c648\") " Jul 21 15:15:58.117993 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117930 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7553ab1-f268-429e-a1e5-e8e4a5c2c648" (UID: "c7553ab1-f268-429e-a1e5-e8e4a5c2c648"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:15:58.117993 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117942 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:15:58.117993 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.117965 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-home" (OuterVolumeSpecName: "home") pod "c7553ab1-f268-429e-a1e5-e8e4a5c2c648" (UID: "c7553ab1-f268-429e-a1e5-e8e4a5c2c648"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:15:58.119906 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.119879 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-dshm" (OuterVolumeSpecName: "dshm") pod "c7553ab1-f268-429e-a1e5-e8e4a5c2c648" (UID: "c7553ab1-f268-429e-a1e5-e8e4a5c2c648"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:15:58.119906 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.119888 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7553ab1-f268-429e-a1e5-e8e4a5c2c648" (UID: "c7553ab1-f268-429e-a1e5-e8e4a5c2c648"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:15:58.120052 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.119942 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-kube-api-access-mnbwq" (OuterVolumeSpecName: "kube-api-access-mnbwq") pod "c7553ab1-f268-429e-a1e5-e8e4a5c2c648" (UID: "c7553ab1-f268-429e-a1e5-e8e4a5c2c648"). InnerVolumeSpecName "kube-api-access-mnbwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:15:58.219476 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.219437 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:15:58.219476 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.219468 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:15:58.219476 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.219479 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbwq\" (UniqueName: \"kubernetes.io/projected/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-kube-api-access-mnbwq\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:15:58.219476 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.219488 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:15:58.219743 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.219497 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7553ab1-f268-429e-a1e5-e8e4a5c2c648-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:15:58.518324 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.518206 2559 generic.go:356] "Generic (PLEG): container finished" podID="c7553ab1-f268-429e-a1e5-e8e4a5c2c648" containerID="7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b" exitCode=0 Jul 21 15:15:58.518324 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.518267 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" Jul 21 15:15:58.518324 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.518287 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" event={"ID":"c7553ab1-f268-429e-a1e5-e8e4a5c2c648","Type":"ContainerDied","Data":"7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b"} Jul 21 15:15:58.518618 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.518338 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn" event={"ID":"c7553ab1-f268-429e-a1e5-e8e4a5c2c648","Type":"ContainerDied","Data":"4da6225e1f6c41a5d5ab2aa71616aeea71e6164cc781334854d17902d7db3203"} Jul 21 15:15:58.518618 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.518363 2559 scope.go:117] "RemoveContainer" containerID="7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b" Jul 21 15:15:58.529639 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.529620 2559 scope.go:117] "RemoveContainer" containerID="7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b" Jul 21 15:15:58.529936 ip-10-0-130-112 kubenswrapper[2559]: E0721 15:15:58.529918 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b\": container with ID starting with 7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b not found: ID does not exist" containerID="7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b" Jul 21 15:15:58.529994 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.529947 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b"} err="failed to get container status \"7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b\": rpc error: code = NotFound desc = could not find container \"7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b\": container with ID starting with 7c634d6d0b3b1082f91112c5701ecbdba4d5663a7cbd476bce9d31f58ae9671b not found: ID does not exist" Jul 21 15:15:58.541815 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.541785 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn"] Jul 21 15:15:58.545133 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.545106 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/auth-test-model-9b06ae-kserve-84cffd48cf-pxsrn"] Jul 21 15:15:58.921245 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:58.921177 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7553ab1-f268-429e-a1e5-e8e4a5c2c648" path="/var/lib/kubelet/pods/c7553ab1-f268-429e-a1e5-e8e4a5c2c648/volumes" Jul 21 15:15:59.308488 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:59.308451 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:59.308696 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:59.308514 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:59.321150 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:59.321121 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:15:59.536001 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:15:59.535969 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:16:30.930324 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:16:30.930281 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:17:17.891943 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:17.891911 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7"] Jul 21 15:17:17.917933 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:17.917902 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:17:18.811682 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:18.811647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:17:25.086739 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.086703 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv"] Jul 21 15:17:25.087191 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.087177 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7553ab1-f268-429e-a1e5-e8e4a5c2c648" containerName="main" Jul 21 15:17:25.087277 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.087192 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7553ab1-f268-429e-a1e5-e8e4a5c2c648" containerName="main" Jul 21 15:17:25.087346 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.087332 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7553ab1-f268-429e-a1e5-e8e4a5c2c648" containerName="main" Jul 21 15:17:25.090798 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.090777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.093606 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.093584 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9b06ae\"/\"rate-test-model-9b06ae-kserve-self-signed-certs\"" Jul 21 15:17:25.100305 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.100281 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv"] Jul 21 15:17:25.113186 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.113152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:17:25.167248 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.167182 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tls-certs\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.167382 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.167292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tmp-dir\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.167382 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.167316 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgcd\" (UniqueName: \"kubernetes.io/projected/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-kube-api-access-jwgcd\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.167382 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.167360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-home\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.167382 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.167380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-model-cache\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.167560 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.167405 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-dshm\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268210 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tmp-dir\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268210 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268210 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgcd\" (UniqueName: \"kubernetes.io/projected/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-kube-api-access-jwgcd\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268502 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-home\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268502 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268290 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-model-cache\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268502 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-dshm\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268502 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tls-certs\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268696 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tmp-dir\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268696 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268670 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-home\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.268770 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.268736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-model-cache\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.270643 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.270620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-dshm\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.270874 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.270857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tls-certs\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.275995 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.275969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgcd\" (UniqueName: \"kubernetes.io/projected/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-kube-api-access-jwgcd\") pod \"rate-test-model-9b06ae-kserve-75bf984d95-nt2dv\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.402423 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.402318 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:25.547039 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.540516 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv"] Jul 21 15:17:25.938655 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.938615 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" event={"ID":"834d2b2a-3a0c-4c20-9c95-a7be78162d6d","Type":"ContainerStarted","Data":"a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a"} Jul 21 15:17:25.938841 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.938663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" event={"ID":"834d2b2a-3a0c-4c20-9c95-a7be78162d6d","Type":"ContainerStarted","Data":"6588c5c937915333c600ccdd02935b286a0340c642c8f834e2b47ba42fc9cb4b"} Jul 21 15:17:25.955870 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:25.955819 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" podStartSLOduration=0.955804961 podStartE2EDuration="955.804961ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:17:25.953697534 +0000 UTC m=+2690.441888794" watchObservedRunningTime="2026-07-21 15:17:25.955804961 +0000 UTC m=+2690.443996188" Jul 21 15:17:32.913685 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:32.913614 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" podUID="930a9075-a663-402d-b0e4-8097443de3c1" containerName="main" containerID="cri-o://4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6" gracePeriod=45 Jul 21 15:17:33.171998 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.171928 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:17:33.252578 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252526 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-home\") pod \"930a9075-a663-402d-b0e4-8097443de3c1\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " Jul 21 15:17:33.252774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252606 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-dshm\") pod \"930a9075-a663-402d-b0e4-8097443de3c1\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " Jul 21 15:17:33.252774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252638 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-model-cache\") pod \"930a9075-a663-402d-b0e4-8097443de3c1\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " Jul 21 15:17:33.252774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252689 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjb4d\" (UniqueName: \"kubernetes.io/projected/930a9075-a663-402d-b0e4-8097443de3c1-kube-api-access-vjb4d\") pod \"930a9075-a663-402d-b0e4-8097443de3c1\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " Jul 21 15:17:33.252774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252722 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-tmp-dir\") pod \"930a9075-a663-402d-b0e4-8097443de3c1\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " Jul 21 15:17:33.252774 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252768 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a9075-a663-402d-b0e4-8097443de3c1-tls-certs\") pod \"930a9075-a663-402d-b0e4-8097443de3c1\" (UID: \"930a9075-a663-402d-b0e4-8097443de3c1\") " Jul 21 15:17:33.253052 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252885 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-home" (OuterVolumeSpecName: "home") pod "930a9075-a663-402d-b0e4-8097443de3c1" (UID: "930a9075-a663-402d-b0e4-8097443de3c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:17:33.253052 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.252910 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "930a9075-a663-402d-b0e4-8097443de3c1" (UID: "930a9075-a663-402d-b0e4-8097443de3c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:17:33.253159 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.253047 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "930a9075-a663-402d-b0e4-8097443de3c1" (UID: "930a9075-a663-402d-b0e4-8097443de3c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:17:33.253268 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.253251 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:17:33.253340 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.253275 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:17:33.253340 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.253290 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:17:33.254904 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.254874 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930a9075-a663-402d-b0e4-8097443de3c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "930a9075-a663-402d-b0e4-8097443de3c1" (UID: "930a9075-a663-402d-b0e4-8097443de3c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:17:33.255020 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.254917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-dshm" (OuterVolumeSpecName: "dshm") pod "930a9075-a663-402d-b0e4-8097443de3c1" (UID: "930a9075-a663-402d-b0e4-8097443de3c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:17:33.255020 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.254934 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930a9075-a663-402d-b0e4-8097443de3c1-kube-api-access-vjb4d" (OuterVolumeSpecName: "kube-api-access-vjb4d") pod "930a9075-a663-402d-b0e4-8097443de3c1" (UID: "930a9075-a663-402d-b0e4-8097443de3c1"). InnerVolumeSpecName "kube-api-access-vjb4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:17:33.354428 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.354379 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a9075-a663-402d-b0e4-8097443de3c1-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:17:33.354428 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.354421 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjb4d\" (UniqueName: \"kubernetes.io/projected/930a9075-a663-402d-b0e4-8097443de3c1-kube-api-access-vjb4d\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:17:33.354428 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.354432 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a9075-a663-402d-b0e4-8097443de3c1-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:17:33.978826 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.978793 2559 generic.go:356] "Generic (PLEG): container finished" podID="930a9075-a663-402d-b0e4-8097443de3c1" containerID="4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6" exitCode=0 Jul 21 15:17:33.979276 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.978841 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" Jul 21 15:17:33.979276 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.978851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" event={"ID":"930a9075-a663-402d-b0e4-8097443de3c1","Type":"ContainerDied","Data":"4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6"} Jul 21 15:17:33.979276 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.978883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7" event={"ID":"930a9075-a663-402d-b0e4-8097443de3c1","Type":"ContainerDied","Data":"e1d14b9f75b1ab8e1b477fc4c2e489da18767c91cc3165e0cdc5caf98ce687f2"} Jul 21 15:17:33.979276 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.978900 2559 scope.go:117] "RemoveContainer" containerID="4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6" Jul 21 15:17:33.987927 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.987910 2559 scope.go:117] "RemoveContainer" containerID="4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6" Jul 21 15:17:33.988199 ip-10-0-130-112 kubenswrapper[2559]: E0721 15:17:33.988179 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6\": container with ID starting with 4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6 not found: ID does not exist" containerID="4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6" Jul 21 15:17:33.988279 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:33.988208 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6"} err="failed to get container status \"4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6\": rpc error: code = NotFound desc = could not find container \"4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6\": container with ID starting with 4e8d1a2546468c1a47164a20fab40dbd688a6213e2cceb76ac6ba5408a5b80d6 not found: ID does not exist" Jul 21 15:17:34.001181 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:34.001147 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7"] Jul 21 15:17:34.004184 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:34.004153 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/sub-test-model-9b06ae-kserve-6d68c76b9-pkwt7"] Jul 21 15:17:34.916826 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:34.916790 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930a9075-a663-402d-b0e4-8097443de3c1" path="/var/lib/kubelet/pods/930a9075-a663-402d-b0e4-8097443de3c1/volumes" Jul 21 15:17:35.403003 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:35.402961 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:35.403003 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:35.403012 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:35.415746 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:35.415718 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:36.001054 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:36.001020 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:17:37.005314 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:37.005278 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:17:37.013630 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:37.013608 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbwvc_e5287c9f-3012-4c0b-bed8-3cd0af5c365b/ovn-acl-logging/0.log" Jul 21 15:17:41.336825 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:41.336789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:17:41.438406 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:17:41.438367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:18:04.624450 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:18:04.624413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:18:08.341755 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:18:08.341712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:18:08.443204 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:18:08.443166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:18:15.833576 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:18:15.833545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:18:15.934069 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:18:15.934034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:18:35.882929 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:18:35.882894 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:18:35.983468 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:18:35.983411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:12.097915 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:12.097673 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv"] Jul 21 15:19:12.118312 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:12.118201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:12.928767 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:12.928735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:18.470533 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:18.470493 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:18.526566 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:18.526531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:27.118271 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.118202 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" podUID="834d2b2a-3a0c-4c20-9c95-a7be78162d6d" containerName="main" containerID="cri-o://a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a" gracePeriod=45 Jul 21 15:19:27.367310 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.367287 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:19:27.458109 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458007 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwgcd\" (UniqueName: \"kubernetes.io/projected/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-kube-api-access-jwgcd\") pod \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " Jul 21 15:19:27.458314 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458109 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tls-certs\") pod \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " Jul 21 15:19:27.458314 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458166 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-model-cache\") pod \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " Jul 21 15:19:27.458314 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458202 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-dshm\") pod \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " Jul 21 15:19:27.458314 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458266 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-home\") pod \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " Jul 21 15:19:27.458537 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tmp-dir\") pod \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\" (UID: \"834d2b2a-3a0c-4c20-9c95-a7be78162d6d\") " Jul 21 15:19:27.458537 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "834d2b2a-3a0c-4c20-9c95-a7be78162d6d" (UID: "834d2b2a-3a0c-4c20-9c95-a7be78162d6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:19:27.458621 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458549 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-home" (OuterVolumeSpecName: "home") pod "834d2b2a-3a0c-4c20-9c95-a7be78162d6d" (UID: "834d2b2a-3a0c-4c20-9c95-a7be78162d6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:19:27.458697 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458673 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "834d2b2a-3a0c-4c20-9c95-a7be78162d6d" (UID: "834d2b2a-3a0c-4c20-9c95-a7be78162d6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:19:27.458752 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458681 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-home\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:19:27.458752 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.458721 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-model-cache\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:19:27.460289 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.460267 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-dshm" (OuterVolumeSpecName: "dshm") pod "834d2b2a-3a0c-4c20-9c95-a7be78162d6d" (UID: "834d2b2a-3a0c-4c20-9c95-a7be78162d6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:19:27.460718 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.460696 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "834d2b2a-3a0c-4c20-9c95-a7be78162d6d" (UID: "834d2b2a-3a0c-4c20-9c95-a7be78162d6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:19:27.460718 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.460703 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-kube-api-access-jwgcd" (OuterVolumeSpecName: "kube-api-access-jwgcd") pod "834d2b2a-3a0c-4c20-9c95-a7be78162d6d" (UID: "834d2b2a-3a0c-4c20-9c95-a7be78162d6d"). InnerVolumeSpecName "kube-api-access-jwgcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:19:27.547016 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.546984 2559 generic.go:356] "Generic (PLEG): container finished" podID="834d2b2a-3a0c-4c20-9c95-a7be78162d6d" containerID="a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a" exitCode=0 Jul 21 15:19:27.547142 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.547033 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" Jul 21 15:19:27.547142 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.547070 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" event={"ID":"834d2b2a-3a0c-4c20-9c95-a7be78162d6d","Type":"ContainerDied","Data":"a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a"} Jul 21 15:19:27.547142 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.547114 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv" event={"ID":"834d2b2a-3a0c-4c20-9c95-a7be78162d6d","Type":"ContainerDied","Data":"6588c5c937915333c600ccdd02935b286a0340c642c8f834e2b47ba42fc9cb4b"} Jul 21 15:19:27.547142 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.547131 2559 scope.go:117] "RemoveContainer" containerID="a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a" Jul 21 15:19:27.555803 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.555788 2559 scope.go:117] "RemoveContainer" containerID="a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a" Jul 21 15:19:27.556046 ip-10-0-130-112 kubenswrapper[2559]: E0721 15:19:27.556027 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a\": container with ID starting with a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a not found: ID does not exist" containerID="a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a" Jul 21 15:19:27.556112 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.556051 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a"} err="failed to get container status \"a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a\": rpc error: code = NotFound desc = could not find container \"a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a\": container with ID starting with a432f235fc8601acba20f56870599ac689564e24abd89c21d0bc03d60f0c9b4a not found: ID does not exist" Jul 21 15:19:27.560032 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.560016 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgcd\" (UniqueName: \"kubernetes.io/projected/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-kube-api-access-jwgcd\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:19:27.560083 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.560036 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tls-certs\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:19:27.560083 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.560047 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-dshm\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:19:27.560083 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.560055 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834d2b2a-3a0c-4c20-9c95-a7be78162d6d-tmp-dir\") on node \"ip-10-0-130-112.ec2.internal\" DevicePath \"\"" Jul 21 15:19:27.567206 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.567186 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv"] Jul 21 15:19:27.570697 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:27.570678 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9b06ae/rate-test-model-9b06ae-kserve-75bf984d95-nt2dv"] Jul 21 15:19:28.919664 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:28.919627 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834d2b2a-3a0c-4c20-9c95-a7be78162d6d" path="/var/lib/kubelet/pods/834d2b2a-3a0c-4c20-9c95-a7be78162d6d/volumes" Jul 21 15:19:32.673673 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:32.673640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:32.841565 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:32.841520 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:42.445373 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:42.445334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:47.094328 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:47.094288 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:47.502732 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:47.502642 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:53.428280 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:53.428239 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:53.493705 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:53.493666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:53.791506 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:53.791479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:53.916846 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:53.916813 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:54.579686 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:54.579653 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:19:54.680608 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:19:54.680575 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:04.472746 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:04.472704 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:07.808730 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:07.808693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:07.910169 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:07.910134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:09.113726 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:09.113692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:09.528551 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:09.528457 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:15.363230 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:15.363196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:15.426456 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:15.426418 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:15.691296 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:15.691174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:15.862019 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:15.861963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:22.614138 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:22.614110 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b978d8c-m8sz5_f1525bf0-4f39-42dd-884a-bcff30e78d12/copy-manifests/0.log" Jul 21 15:20:22.621845 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:22.621821 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b978d8c-m8sz5_f1525bf0-4f39-42dd-884a-bcff30e78d12/manager/0.log" Jul 21 15:20:25.082235 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:25.082186 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-b4pzh_482162f6-cb38-460a-bd48-17e37b8e01ed/manager/0.log" Jul 21 15:20:26.606295 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:26.606257 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d9d9b8c-j9xq5_50731338-efa4-44f8-9c26-cd899e23915a/kube-auth-proxy/0.log" Jul 21 15:20:26.866233 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:26.866125 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-s4tx5_64cae0ef-d6e1-4d17-abbd-16dfdfda4a73/payload-pre-processing/0.log" Jul 21 15:20:27.585834 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:27.585748 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lrs6x_49ecc370-f3c6-4120-a9e6-402a6d5c920b/main/0.log" Jul 21 15:20:27.839150 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:27.839065 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsf4nb_302bf22c-580e-4cbb-8dff-542048009696/main/0.log" Jul 21 15:20:27.967555 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:27.967524 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vdn7d_a21a0d36-a578-4628-af44-6714884a76ce/main/0.log" Jul 21 15:20:28.115638 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:28.115538 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bnqt2_153a5b47-c9c5-46c7-980d-18bdb5b264c5/main/0.log" Jul 21 15:20:33.783521 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:33.783489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:33.883746 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:33.883713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" limit=1024 Jul 21 15:20:38.133553 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:38.133516 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wpdjf_92704e7d-742e-4358-a920-286027d064b0/global-pull-secret-syncer/0.log" Jul 21 15:20:38.202733 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:38.202703 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lc8kv_420a38b5-47bb-4326-927f-cc0a80bde4c4/konnectivity-agent/0.log" Jul 21 15:20:38.317992 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:38.317958 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-112.ec2.internal_1d033722713bf948e6f973d6228bc956/haproxy/0.log" Jul 21 15:20:43.700208 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:43.700165 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-b4pzh_482162f6-cb38-460a-bd48-17e37b8e01ed/manager/0.log" Jul 21 15:20:45.999652 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:45.999620 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de6bf28c-c86b-4b74-8b8c-ba96fda955b2/alertmanager/0.log" Jul 21 15:20:46.022289 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.022262 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de6bf28c-c86b-4b74-8b8c-ba96fda955b2/config-reloader/0.log" Jul 21 15:20:46.044076 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.044050 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de6bf28c-c86b-4b74-8b8c-ba96fda955b2/kube-rbac-proxy-web/0.log" Jul 21 15:20:46.066097 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.066075 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de6bf28c-c86b-4b74-8b8c-ba96fda955b2/kube-rbac-proxy/0.log" Jul 21 15:20:46.087754 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.087709 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de6bf28c-c86b-4b74-8b8c-ba96fda955b2/kube-rbac-proxy-metric/0.log" Jul 21 15:20:46.109076 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.109058 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de6bf28c-c86b-4b74-8b8c-ba96fda955b2/prom-label-proxy/0.log" Jul 21 15:20:46.132479 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.132456 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de6bf28c-c86b-4b74-8b8c-ba96fda955b2/init-config-reloader/0.log" Jul 21 15:20:46.227800 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.227736 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-765dl_7db794aa-bce8-4c7a-a0ec-32c49123856b/kube-state-metrics/0.log" Jul 21 15:20:46.244764 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.244740 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-765dl_7db794aa-bce8-4c7a-a0ec-32c49123856b/kube-rbac-proxy-main/0.log" Jul 21 15:20:46.267126 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.267103 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-765dl_7db794aa-bce8-4c7a-a0ec-32c49123856b/kube-rbac-proxy-self/0.log" Jul 21 15:20:46.292119 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.292098 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66b5c874f-8fb4d_9c4d2521-bed9-4fbc-bf84-dbca7d7c5c42/metrics-server/0.log" Jul 21 15:20:46.316314 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.316294 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-bqmlj_733c0726-a082-4800-b7c1-4442623b4d99/monitoring-plugin/0.log" Jul 21 15:20:46.345994 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.345976 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwqsw_18e8d538-3244-4033-a8d7-521344c52f96/node-exporter/0.log" Jul 21 15:20:46.369874 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.369855 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwqsw_18e8d538-3244-4033-a8d7-521344c52f96/kube-rbac-proxy/0.log" Jul 21 15:20:46.390062 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.390044 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cwqsw_18e8d538-3244-4033-a8d7-521344c52f96/init-textfile/0.log" Jul 21 15:20:46.565845 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.565814 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gqjgm_61e17f7b-b271-4f20-976b-b90048f33b92/kube-rbac-proxy-main/0.log" Jul 21 15:20:46.586201 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.586167 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gqjgm_61e17f7b-b271-4f20-976b-b90048f33b92/kube-rbac-proxy-self/0.log" Jul 21 15:20:46.607702 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.607671 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gqjgm_61e17f7b-b271-4f20-976b-b90048f33b92/openshift-state-metrics/0.log" Jul 21 15:20:46.646432 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.646397 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a11c2ef6-45ad-4830-b9d8-2835955866d3/prometheus/0.log" Jul 21 15:20:46.666267 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.666238 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a11c2ef6-45ad-4830-b9d8-2835955866d3/config-reloader/0.log" Jul 21 15:20:46.693419 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.693393 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a11c2ef6-45ad-4830-b9d8-2835955866d3/thanos-sidecar/0.log" Jul 21 15:20:46.718330 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.718300 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a11c2ef6-45ad-4830-b9d8-2835955866d3/kube-rbac-proxy-web/0.log" Jul 21 15:20:46.742115 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.742088 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a11c2ef6-45ad-4830-b9d8-2835955866d3/kube-rbac-proxy/0.log" Jul 21 15:20:46.773173 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.773144 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a11c2ef6-45ad-4830-b9d8-2835955866d3/kube-rbac-proxy-thanos/0.log" Jul 21 15:20:46.793759 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.793699 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a11c2ef6-45ad-4830-b9d8-2835955866d3/init-config-reloader/0.log" Jul 21 15:20:46.821576 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.821471 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-z24cv_52ea19f8-d5ea-4ebd-a85a-7596f091f628/prometheus-operator/0.log" Jul 21 15:20:46.838331 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.838299 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-z24cv_52ea19f8-d5ea-4ebd-a85a-7596f091f628/kube-rbac-proxy/0.log" Jul 21 15:20:46.877605 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.877570 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-hx8jh_94aafa15-2b91-4c73-b51b-e622db2bf7c3/prometheus-operator-admission-webhook/0.log" Jul 21 15:20:46.908812 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.908778 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ccb7d6cb4-rlnd2_3b08755e-797d-4094-8bf9-e0f78356bea6/telemeter-client/0.log" Jul 21 15:20:46.931093 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.931065 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ccb7d6cb4-rlnd2_3b08755e-797d-4094-8bf9-e0f78356bea6/reload/0.log" Jul 21 15:20:46.953500 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.953467 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-ccb7d6cb4-rlnd2_3b08755e-797d-4094-8bf9-e0f78356bea6/kube-rbac-proxy/0.log" Jul 21 15:20:46.983995 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:46.983965 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ff49f758d-2f4qx_a9f908db-11df-4eee-bc22-1dc0a79bb931/thanos-query/0.log" Jul 21 15:20:47.006622 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.006593 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ff49f758d-2f4qx_a9f908db-11df-4eee-bc22-1dc0a79bb931/kube-rbac-proxy-web/0.log" Jul 21 15:20:47.029449 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.029411 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ff49f758d-2f4qx_a9f908db-11df-4eee-bc22-1dc0a79bb931/kube-rbac-proxy/0.log" Jul 21 15:20:47.051826 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.051793 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ff49f758d-2f4qx_a9f908db-11df-4eee-bc22-1dc0a79bb931/prom-label-proxy/0.log" Jul 21 15:20:47.072213 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.072131 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ff49f758d-2f4qx_a9f908db-11df-4eee-bc22-1dc0a79bb931/kube-rbac-proxy-rules/0.log" Jul 21 15:20:47.093168 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.093134 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-ff49f758d-2f4qx_a9f908db-11df-4eee-bc22-1dc0a79bb931/kube-rbac-proxy-metrics/0.log" Jul 21 15:20:47.992805 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.992773 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j"] Jul 21 15:20:47.993188 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.993168 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834d2b2a-3a0c-4c20-9c95-a7be78162d6d" containerName="main" Jul 21 15:20:47.993188 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.993181 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="834d2b2a-3a0c-4c20-9c95-a7be78162d6d" containerName="main" Jul 21 15:20:47.993392 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.993197 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930a9075-a663-402d-b0e4-8097443de3c1" containerName="main" Jul 21 15:20:47.993392 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.993202 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="930a9075-a663-402d-b0e4-8097443de3c1" containerName="main" Jul 21 15:20:47.993392 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.993356 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="930a9075-a663-402d-b0e4-8097443de3c1" containerName="main" Jul 21 15:20:47.993392 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.993379 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="834d2b2a-3a0c-4c20-9c95-a7be78162d6d" containerName="main" Jul 21 15:20:47.996684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.996667 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:47.999348 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.999310 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scg49\"/\"openshift-service-ca.crt\"" Jul 21 15:20:47.999516 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:47.999490 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-scg49\"/\"default-dockercfg-bd55p\"" Jul 21 15:20:48.000285 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.000271 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scg49\"/\"kube-root-ca.crt\"" Jul 21 15:20:48.005164 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.005141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j"] Jul 21 15:20:48.135341 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.135304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwm87\" (UniqueName: \"kubernetes.io/projected/8694dfa1-0e77-4844-95e0-810d00ed26ab-kube-api-access-gwm87\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.135720 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.135359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-sys\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.135720 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.135383 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-proc\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.135720 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.135511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-podres\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.135720 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.135544 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-lib-modules\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236433 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-podres\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236433 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-lib-modules\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwm87\" (UniqueName: \"kubernetes.io/projected/8694dfa1-0e77-4844-95e0-810d00ed26ab-kube-api-access-gwm87\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-sys\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-proc\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-podres\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236594 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-sys\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236684 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-proc\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.236956 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.236687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8694dfa1-0e77-4844-95e0-810d00ed26ab-lib-modules\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.244117 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.244056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwm87\" (UniqueName: \"kubernetes.io/projected/8694dfa1-0e77-4844-95e0-810d00ed26ab-kube-api-access-gwm87\") pod \"perf-node-gather-daemonset-dwp2j\" (UID: \"8694dfa1-0e77-4844-95e0-810d00ed26ab\") " pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.307796 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.307774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.436406 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.436377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j"] Jul 21 15:20:48.438298 ip-10-0-130-112 kubenswrapper[2559]: W0721 15:20:48.438268 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8694dfa1_0e77_4844_95e0_810d00ed26ab.slice/crio-e5d46f76fd48da87fa93ccd3edb2931976544a2d9f207dbb682ade920a370f9a WatchSource:0}: Error finding container e5d46f76fd48da87fa93ccd3edb2931976544a2d9f207dbb682ade920a370f9a: Status 404 returned error can't find the container with id e5d46f76fd48da87fa93ccd3edb2931976544a2d9f207dbb682ade920a370f9a Jul 21 15:20:48.936429 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.936395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" event={"ID":"8694dfa1-0e77-4844-95e0-810d00ed26ab","Type":"ContainerStarted","Data":"38c4cba27237eed61d31a6817400d40cd4f89d0f380e315208e3645dec066270"} Jul 21 15:20:48.936596 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.936444 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" event={"ID":"8694dfa1-0e77-4844-95e0-810d00ed26ab","Type":"ContainerStarted","Data":"e5d46f76fd48da87fa93ccd3edb2931976544a2d9f207dbb682ade920a370f9a"} Jul 21 15:20:48.936596 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.936527 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:48.953560 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:48.953516 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" podStartSLOduration=1.953504283 podStartE2EDuration="1.953504283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 15:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:20:48.952978222 +0000 UTC m=+2893.441169427" watchObservedRunningTime="2026-07-21 15:20:48.953504283 +0000 UTC m=+2893.441695487" Jul 21 15:20:51.171683 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:51.171649 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tfp45_bd5ce839-4b3a-4927-a991-1aa5cfad8af1/dns/0.log" Jul 21 15:20:51.193044 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:51.193012 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tfp45_bd5ce839-4b3a-4927-a991-1aa5cfad8af1/kube-rbac-proxy/0.log" Jul 21 15:20:51.243811 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:51.243744 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wkxm_2e9595f3-e87e-46b8-993b-9fb4981d50d8/dns-node-resolver/0.log" Jul 21 15:20:51.838604 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:51.838564 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9mjkx_96be4230-5b49-4819-b027-1c57fc60964d/node-ca/0.log" Jul 21 15:20:53.098711 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:53.098677 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d9d9b8c-j9xq5_50731338-efa4-44f8-9c26-cd899e23915a/kube-auth-proxy/0.log" Jul 21 15:20:53.225731 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:53.225700 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-s4tx5_64cae0ef-d6e1-4d17-abbd-16dfdfda4a73/payload-pre-processing/0.log" Jul 21 15:20:53.959428 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:53.959397 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7rjrn_130ab0a9-4626-4c0a-adcf-e715b2cf52ea/serve-healthcheck-canary/0.log" Jul 21 15:20:54.622090 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:54.622060 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r9s6p_4352a4a4-059c-433d-ba11-81053151750c/kube-rbac-proxy/0.log" Jul 21 15:20:54.643189 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:54.643165 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r9s6p_4352a4a4-059c-433d-ba11-81053151750c/exporter/0.log" Jul 21 15:20:54.668036 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:54.667986 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r9s6p_4352a4a4-059c-433d-ba11-81053151750c/extractor/0.log" Jul 21 15:20:54.950792 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:54.950721 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-scg49/perf-node-gather-daemonset-dwp2j" Jul 21 15:20:57.643026 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:57.642998 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b978d8c-m8sz5_f1525bf0-4f39-42dd-884a-bcff30e78d12/manager/0.log" Jul 21 15:20:57.668304 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:57.668279 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b978d8c-m8sz5_f1525bf0-4f39-42dd-884a-bcff30e78d12/copy-manifests/0.log" Jul 21 15:20:58.752013 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:58.751927 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d9d9b8c-j9xq5_50731338-efa4-44f8-9c26-cd899e23915a/kube-auth-proxy/0.log" Jul 21 15:20:58.861927 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:58.861892 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-s4tx5_64cae0ef-d6e1-4d17-abbd-16dfdfda4a73/payload-pre-processing/0.log" Jul 21 15:20:59.043519 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:20:59.043473 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77cc7cb7fc-tqm5z_f75dd4fa-b5ce-4576-a202-54038ecce86e/manager/0.log" Jul 21 15:21:00.400354 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:21:00.400323 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d9d9b8c-j9xq5_50731338-efa4-44f8-9c26-cd899e23915a/kube-auth-proxy/0.log" Jul 21 15:21:00.508896 ip-10-0-130-112 kubenswrapper[2559]: I0721 15:21:00.508870 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-s4tx5_64cae0ef-d6e1-4d17-abbd-16dfdfda4a73/payload-pre-processing/0.log"